ESP8266 Community Forum

- Mon Sep 12, 2016 12:59 pm #54968 OK - as an exercise I wrote a simple program to convert a hex string to an integer. (Yes, I know there's an internal function to do this!) The program seemed to run fine, but I started getting some odd results. This is the code:

Code: Select allinstr$ = ""

do

  do
    if serial.available() > 0 then instr$ = instr$ & serial.read.chr()
  loop until right(instr$,1) = chr(13)

  if left(instr$, 1) = "*" then
    serialflush
    end
  else
    instr$ = left(instr$, len(instr$) - 1)
  end if

  gosub [strtohex]
  print "Value = " & str(rval)
loop while rval <> 1


[strtohex]
rval = 0
if left(instr$, 2) = "0x" then instr$ = mid(instr$, 3, 99)
instr$ = upper(instr$)

for i = 1 to len(instr$)
  t = asc(mid(instr$, i, 1))

  if t > 57 then
    t = t - 55
  else
    t = t - 48
  end if

  rval = rval * 16 + t
next i

instr$ = ""
return



No, it's not the neatest but, like I say, it was an exercise to test various functions.

Anyway, all seemed to go well; put in FABC and get 64188. Put in FABCD and get 1027021.
Put in FABCDEF and get 262917616 - oops! That should be 262917615.

Problem is, put in anything between FABCDE9 and FABCDEF and you get the same result. Put in anything between FABCDE0 and FABCDE8 and you get 262917600 for each.

Because I thought my logic might be screwy I put in the line

Code: Select all  print "HexToInt gives  " & hextoint(instr$)

immediately before the gosub - and got exactly the same result with that!

Now, maybe there's a limit on integer values that was unaware of, but in this case, why did the answer 'flip up' by 16 at the 8/9 point?

Anyone got any thoughts?

Phil

- Tue Sep 13, 2016 3:16 am #55003 Without digging or reading your code, I'd say you were probably right and falling over the limits for size of a numeric variable in espbasic. I'm not sure what these are, or the limits of accuracy in the code which underlies espbasic. try googling two's complement - it may give some hints. Where there's only one numeric type, remember it has to cope with sign and decimal places as well...