ESP8266 Community Forum

- Fri May 01, 2015 1:44 pm #16217 Cal,

The issue with stings is a combination of two aspects:

• all of the string routines that need to use a temporary working string declare a luaL_Buffer xx;and this allocates a fixed char buffer[LUAL_BUFFERSIZE] on the local stack.
• because routines like string.gsub() can introduce call-backs which then call more string manipulation, you can generated multiple of these LUAL_BUFFERSIZE on the stack.

The developers have played with LUAL_BUFFERSIZE options: 4Kb and 1Kb and you've just proposed 256b. But from an apps programmer perspective you are caught in a dilemma: you can't generate any string intermediary longer than LUAL_BUFFERSIZE as this will cause a panic, but if its too large then you can run out of heap.

I personally think that 256b is too small. I can live with 1K, but would prefer 2K, because I know how to code to avoid nesting string calls too deep. The alternative of moving the buffer into the heap would require a lot of code changes and would grow the size of lstrilib.c.

Please see my Developer FAQ for Q&A on nodeMCU Lua before asking a question here that's already been answered. This and the other official documentation should be your first reference for developing Lua Apps

- Fri May 01, 2015 2:08 pm #16219

Cal,

The issue with stings is a combination of two aspects:

• all of the string routines that need to use a temporary working string declare a luaL_Buffer xx;and this allocates a fixed char buffer[LUAL_BUFFERSIZE] on the local stack.
  

Right.

because routines like string.gsub() can introduce call-backs which then call more string manipulation, you can generated multiple of these LUAL_BUFFERSIZE on the stack.

Right.

The developers have played with LUAL_BUFFERSIZE options: 4Kb and 1Kb and you've just proposed 256b. But from an apps programmer perspective you are caught in a dilemma: you can't generate any string intermediary longer than LUAL_BUFFERSIZE as this will cause a panic

Thats an unfounded claim. Why should it? Take a look at

Code: Select all#define luaL_addchar(B,c) \
  ((void)((B)->p < ((B)->buffer+LUAL_BUFFERSIZE) || luaL_prepbuffer(B)), \
   (*(B)->p++ = (char)(c)))

LUALIB_API char *luaL_prepbuffer (luaL_Buffer *B) {
  if (emptybuffer(B))
    adjuststack(B);
  return B->buffer;
}

static int emptybuffer (luaL_Buffer *B) {
  size_t l = bufflen(B);
  if (l == 0) return 0;  /* put nothing on stack */
  else {
    lua_pushlstring(B->L, B->buffer, l);
    B->p = B->buffer;
    B->lvl++;
    return 1;
  }
}


If buffer is exhausted its value is pushed on the lua stack aka heap and buffer pointer is reset.
I think its a good solution.

, but if its too large then you can run out of heap.

Whats your point? If you create a string too long for memory you exhaust memory?


4K was a bad idea with a stack size of less <8.5k. I will propose 256 bytes but have to dig up the
quote of the lua author saying its ok.

I personally think that 256b is too small. I can live with 1K, but would prefer 2K, because I know how to code to avoid nesting string calls too deep. The alternative of moving the buffer into the heap would require a lot of code changes and would grow the size of lstrilib.c.

Not needed at all. Please look at the code again.

Get your modules here:

https://www.banggood.com/custlink/KvGGhGF4wG

- Fri May 01, 2015 3:32 pm #16223 Cal, thanks for pointing this out. Yes, I had missed it. Sorry. But to be honest, I've looked through the code and it's as clear as mud. It's a good example of the FLOSS zero-commenting "because the code says it all" fails miserably. To get a better handle, I'd really need to to step through this with gdb. (if we have gdb remote for this system working).

If it does what I think you are implying then it broadly implements the sort of strategy that I've used myself and seen in other applications:

• Initially allocate X bytes on the stack, but it this overflows then realloc a working copy on the heap and realloc this as necessary.
• This way 95% of string buffer manipulation is cheap in terms of runtime cost but the code will handle longer strings.
• However, this isn't using standard malloc / free but is drawing down on a LIFO string stack.

Is this how you read it? If this summary is correct, then this initial allocation constant could easily be dropped to 128b, say. However this leads to a second issue: LUAL_BUFFERSIZE is denormalised in that it is used for two quiet separate purposes: one is to defined this default stack allocated size of a luaL_Buffer, and the second is in (file.c and liolib.c where it is used for allocating I/O buffers and defining I/O chunk sizes, which really need to be aligned to optimum flash write blocksizes.

None of thes other uses seem to constrain functionality, yet I cant' square this observation with the issues which seemed to be cured by an x4 on this size.

Please see my Developer FAQ for Q&A on nodeMCU Lua before asking a question here that's already been answered. This and the other official documentation should be your first reference for developing Lua Apps

- Fri May 01, 2015 4:00 pm #16226

Cal, thanks for pointing this out. Yes, I had missed it. Sorry. But to be honest, I've looked through the code and it's as clear as mud. It's a good example of the FLOSS zero-commenting "because the code says it all" fails miserably. To get a better handle, I'd really need to to step through this with gdb. (if we have gdb remote for this system working).


If it does what I think you are implying then it broadly implements the sort of strategy that I've used myself and seen in other applications:

• Initially allocate X bytes on the stack, but it this overflows then realloc a working copy on the heap and realloc this as necessary.
  [/quote/]Ack.

• This way 95% of string buffer manipulation is cheap in terms of runtime cost but the code will handle longer strings.
  

Ack.

However, this isn't using standard malloc / free but is drawing down on a LIFO string stack.

Nack. I think. Seems to end in standard alloc code.

Is this how you read it? If this summary is correct, then this initial allocation constant could easily be dropped to 128b, say.

Yes. But we need to check other uses as you say.

However this leads to a second issue: LUAL_BUFFERSIZE is denormalised in that it is used for two quiet separate purposes: one is to defined this default stack allocated size of a luaL_Buffer, and the second is in (file.c and liolib.c where it is used for allocating I/O buffers and defining I/O chunk sizes, which really need to be aligned to optimum flash write blocksizes.

None of thes other uses seem to constrain functionality, yet I cant' square this observation with the issues which seemed to be cured by an x4 on this size.

Don't get you?
I will check the other uses again. We should check if the change from 4K to 1K needs to be undone for the other uses, too. It smells like we should add a constant at the same place where LUAL_BUFFERSIZE is defined which
has some I/O purpose. May the code lead us ...

Get your modules here:

https://www.banggood.com/custlink/KvGGhGF4wG