ESP8266 Community Forum

- Fri May 01, 2015 9:56 am #16188 Hello to the nodemcu world!

after hunting some bugs and stability problems of current nodemcu dev branch down I came to
the conclusion that too many of them seem to be solved when using a newer SDK 1.0.X.
The current dev branch is based on SDK 0.9.5.

Here are just a few:

• http://www.esp8266.com/viewtopic.php?f=18&t=1418
• http://www.esp8266.com/viewtopic.php?f=18&t=2739
• https://github.com/nodemcu/nodemcu-firmware/issues/362
• https://github.com/nodemcu/nodemcu-firmware/issues/360

I started with fork https://github.com/chadouming/nodemcu-firmware by chad cormier roussel (chadouming),
which seems to be ( commit 7755a48382a8ef12c0022f492d7c1c5be5178084):

• based on dev096 branch of official nodemcu repo
• some module deactivated
• a few Makefile fixes
• SDK 1.0 libs (I didn't checked which version)

Unfortunately the binary libs of the SDK need about 4K more data memory which means 17K instead of 21K free heap
space in lua.

This is mostly BSS in the following files: (left: 0.9.5, right 1.0 from above repository, first 4 byte value is size in hex)

Code: Select alllibat.a, at_port.o:
  3 .bss          00000a9c  00000000  00000000  00000040  2** |   3 .bss          00000e0c  00000000  00000000  00000040  2**
libpp.a, wdev.o:
  4 .bss          000041c0  00000000  00000000  00000ef0  2** |   4 .bss          00004e80  00000000  00000000  00001200  2**


I think we cannot ignore SDK 1.0 if don't want to track down already fixed bugs over and over again.

Compiler and linker optimization I can think of seem to be already done.
Some data may be movable to rodata as was already done for some other.

Making it easier for people to customize lua firmware to their needs is one way. As I understood it TerryE works on this.

Strings (some? all?) are already moved to read-only section.

I wonder if we can reduce storage requirement of lua programs if we try to reuse the read only stored strings
when parsing lua programs. (Like using an identity map in java.)
That way the lua modules and programs can use long meaningful names without needing heap memory.

Storing the code of modules in flash is another option (TODO: link to thread needed).

I would like to collect ideas and results of experiments here and probably move them to the wiki
it it seems useful.

What do you think?

Cal

Get your modules here:

https://www.banggood.com/custlink/KvGGhGF4wG

- Fri May 01, 2015 10:34 am #16193 I need to add the extra logic into ChunkySpy to decode the eLua tags so that I can look at how the eLua generated code is different from standard lua, but having done a size comparison of chunk sizes on on both my Laptop and the ESP8266, three's not that much. But the VM stores lineNo and variable info for diagnostic / error reporting in a way that doubles the size of a chunk for a given routine and this is a major component of the RAM overhead. So this needs looking at.

Also AFAIK, the code itself (and ROM tables) is sort of VM paged from flash, so they don't take up too much space, but all of the modules take up some RAM after initialisation. Another thought is that this initialisation could be done on a lazy basis so you only initialise the modules that you are using and the others don't take up RAM.

There's also basic 101 stuff that you can do on the Apps side:

• always use LuaSrcDiet to source compress any modules that you download to the ESP, (except maybe if you in the process of developing/debugging one.
• Compile and modules that you use in your basic processing loops.
• Use locals wherever possible
• Make sure you understand the Lua scoping rules and GC algo to make sure that you derefernce and GC all dead stuff
• Don't be afraid to make all of your core processing ephemeral -- Loading compiled code from flash takes of the order of 1 mSec
• Understand the stupid way the internal string buffer allocation works and how any string-based callbacks can kill you.
• Don't be afraid to reboot the ESP between requests for the ultimate clear down.

My framework isn't usable by 3rd parties yet, but I can operate my ESP8266 including development over the Wifi for a 3.5kb framework overhead (including the autoloader).

I need to ponder on the wider issues that you raise.

Please see my Developer FAQ for Q&A on nodeMCU Lua before asking a question here that's already been answered. This and the other official documentation should be your first reference for developing Lua Apps

- Fri May 01, 2015 11:01 am #16199 Since we are talking 1.0, here is my experience in the last few days.

1.0.1 is missing many headers that 1.0.0 has and lib{c,hal}.a are also not in the archive. I copied these across and could compile my project. It "mostly" worked, only issue is that sent packets failed. I get a disconnect about 3s after the send and never get a 'sent'.

Now maybe the missing files are really not needed and some code change is required to not refer to them? Don't know.
Maybe 1.0.1 should be extracted on top of 1.0.0?

Size-wise I had no problems (I do remove most modules and add a small one though). However the failed send could actually be an indication of a failure due to memory shortage (not unheard). Compared to 0.9.5 I see about 2KB less heap available.

cheers

- Fri May 01, 2015 11:10 am #16201

Also AFAIK, the code itself (and ROM tables) is sort of VM paged from flash, so they don't take up too much space, but all of the modules take up some RAM after initialisation.

The CPU has a kind of harvard architecture. That means data and code are separated areas in RAM.
There is no VM (virtual memory?) involved IMHO. The code in flash memory is backed by some cache.
Code size is currently no big issue.
Stack size is a separate issue but it does not conflict with heap/data. It seems to be OK to lower the string handling buffer size from 1K to 256 bytes. I need to check string handling again. Because of the heap pressure I suggest
to not change the string handling code to malloc.
Data size is the real problem. Variables of modules and heap share the same memory.

Another thought is that this initialisation could be done on a lazy basis so you only initialise the modules that you are using and the others don't take up RAM.

Good idea.

[*]Understand the stupid way the internal string buffer allocation works and how any string-based callbacks can kill you.

I don't agree 100% here.
With lower buffer size the stack based string handling part seems to OK and keeps some pressure away from the heap.
I found out that maximum stack size is even lower than I tought last time. <8.5 K
The lua runtime does not have checks in place so you are able to overrun the stack that results in
spontaneous reboots. I have code that checks for this buts its not a 100% solution.

Get your modules here:

https://www.banggood.com/custlink/KvGGhGF4wG