lua: lnum: fix strtoul based number parsing

Lua's LNUM patch currently doesn't parse properly certain numbers as
it's visible from the following simple tests.

On x86_64 host (stock Lua 5.1.5, expected output):

 $ /usr/bin/lua -e 'print(0x80000000); print(0x80000000000); print(0x100000000)'

  2147483648
  8796093022208
  4294967296

On x86_64 host:

 $ staging_dir/hostpkg/bin/lua -e 'print(0x80000000); print(0x80000000000); print(0x100000000)'

  -2147483648
  0
  0

On x86_64 target:

 $ lua -e 'print(0x80000000); print(0x80000000000); print(0x100000000)'

  -2147483648
  0
  0

On ath79 target:

 $ lua -e 'print(0x80000000); print(0x80000000000); print(0x100000000)'

  -2147483648
  8796093022208
  4294967296

It's caused by two issues fixed in this patch, first issue is caused by
unhadled strtoul overflow and second one is caused by the cast of
unsigned to signed Lua integer when parsing from hex literal.

Run tested on:

 * Zidoo Z9S with RTD1296 CPU (aarch64_cortex-a53)
 * qemu/x86_64
 * qemu/armvirt_64
 * ath79

Signed-off-by: Liangbin Lian <jjm2473@gmail.com>
[commit subject/message touches, fixed From to match SOB, fixed another
 unhandled case in luaO_str2i, host Lua, package bump]
Signed-off-by: Petr Štetiar <ynezz@true.cz>