ESP8266 Community Forum

- Thu Apr 02, 2015 8:53 am #13274 Hi, since this is my first post, let me introduce myself. My name is Terry Ellison and my handle is TerryE; I have 30+ years in IT, and maybe 12 years contributing to Open Source projects. I have contributed to OpenOffice.org including the Calc Basic engine; I also set up and was admin for the OOo User forums (phpBB3), and was admin on the OOo User Wiki (Mediawiki) -- until Apache took over, and I got fed up with how they wanted to run things. I also helped set up and was moderator on the Virtual Box forum (phpBB3) and did a few contributions here. I have also contributed to the PHP internal Zend engine and its Opcache. I am still active on Wiipedia. Just look for my handle on any of these, if you are interested. This is only background context for some of my comments below.

I am currently in the process of building my own house (see my blog on ebuild.co.uk) and this is taking up most of my free time, but I am looking at ESP8266-based solutions for Home Automation, and I decided to go the Lua route. Hence it was time to get to grips with yet another language, the NodeMCU eLua variant, and because of my previous interests, I've also been taking a deep-dive into the NodeMCU internals. Overall, I am extremely impressed with eLua, the ESP 8266 and how the NodeMCU people have integrated these into an excellent offering.

The Lua runtime system (RTS) is in many ways similar to that of PHP, but its also a lot leaner and faster, and the eLua variant seems to be extremely well tuned to this type of embedded application. Nonetheless, application developers do have to work within the system constraints and in particular:

• The available application RAM is just under 22K on the NodeMCU 0.9.5 build
• The file system space is under 60K on the 512Kb Flash variants like the ESP-01

This is small enough that developers need to understand the standard techniques for squeezing applications into this sort of space, and I would have expected to see them documented here and easy-to-understand guidelines or on the wiki. However, I have a scan and not found them, but I have found enough by confused developers who are hitting these constraints. So it does seem that we need some such guidelines on the wiki, and I would be happy to propose a strawman set either here or on the wiki if there is sufficient interest.

An example of the sort of thing that I am talking about is that developers should have some understanding of the following:

• Lua uses a Mark and Sweep Garbage Collector which is extremely well suited to this small-memory type of RTS, and that fully dereferencing any object or string within the application will make it memory available after the next GC sweep
• Lua is an incremental compile system (like PHP), but unlike PHP the opcode arrays are a lot more compact (a typical line of Lua will generate maybe 3 or so 4-btye instructions. Once compiled these are stored in RAM, but unlike PHP the opcode arrays are just another object as far as GC is concerned, so the memory can be reclaimed for other use if properly dereferenced.
• The Lua RTS treats compiled and source files as largely interchangeable. The only difference is that looks for a standard <ESC>Lua header as the first 4 bytes and if found it uses the (very fast) oparray loader to load code from Flash instead of invoking the compiler. So the ".lua" or ".lc" extension is just user-sugar as this snippet shows, you can even have a compiled init.lua
  Code: Select allfile.open("test.lua","w")
file.writeline([[collectgarbage();print("Compile:",node.heap()); file.remove("init.lua")]])
file.close()
node.compile("test.lua")
file.remove("init.lua"); file.rename("test.lc","init.lua")
node.restart()

• Loading compiled functions from Flash is pretty fast (~1mSec). ( Those who know C have a look at app/lua/lundump.c which does this.) So you can do tricks such as using a setmetatable to declare on __index autoloader, and this gives you the ability to pretty transparently load methods in from Flash (and GC them when done) with minimal performance impact. This enables you to fit pretty big applications into the 21Kb RAM limit. (I am developing a little framework based on this but this is the subject of a separate topic.

I could go on but before I do, I just want to pause for comments / feedback / reactions. Thanks

Please see my Developer FAQ for Q&A on nodeMCU Lua before asking a question here that's already been answered. This and the other official documentation should be your first reference for developing Lua Apps

- Fri Apr 03, 2015 8:31 am #13356 Moin Terry!

I would definitely highly appreciate information like that.

I am highly interested in making the nodemcu platform more robust even on an esp 01.
Currently I try try get my head around the memory model used (stack versus heap) the
constraints that sets for the lua and the C code and getting better debug information in case of errors
that lead to reboots. Replacing/wrapping most lua_call with lua_pcalls looks promising for me.

Using the flash as a kind of shared library pool sounds cool.

Seeing more and more people adding modules on the C level I wonder if something similiar would be
possible using C modules more one step after the other.

Carsten

P.S.
Note to other esp8266/nodemcu enthusiasts just getting started: Don't get discouraged if working with people having
a long vita in the IT world. Talking for myself I can say that even with 30+ years doing programming on systems
of increasing size programming in lua on such a small system makes me wear a white belt again.
Thanks to the espressif team and the nodemcu team for creating such a fun item.
Sharing knowledge and organizing things makes this forum a great place to be! I learned a lot from examples,
questions and answers here, thank you very much!
Like bit banging the PET user port for teletype printing again

Get your modules here:

https://www.banggood.com/custlink/KvGGhGF4wG

- Fri Apr 03, 2015 7:24 pm #13414 Carsten,

I see there being two facets here: one is a detailed technical discussion so that the relevant techies can come to a common understanding, and then we write this up in an accessible format for the apps devs who don't want to have to get into the nodeMCU eLua internals to use. The wiki is probably the best place for this.

A couple of examples of these sorts of issues. I recall you were wondering about how the stack intersected with the heap in another post (I don't have enough karma yet to search for it). Well lua doesn't really have a stack. Everything is a heap object. Because of lua semantics, a function has a known number of instructions, constants, locals, up values, which is defined in the Proto block for the function. When your code issues a call, the VM just allocs the correctly sized raw block from the heap to hold all of this, and when you return, this is freed. So there isn't a conventional stack and SP.

I also mentioned this issue of freeing up code. Yes, this does occur when functions are properly dereferenced, but this doesn't happen when you might think it does. Consider the example

Code: Select alllocal x =function()  ... end
...
x=nil

Setting x to nil doesn't free up anything. The assignment binds a closure to a local value, but the enclosure is linked to a hidden function reference in the defining function. You need to free up the latter, to free up the functions. In practice this means that when you assign a load file, etc., to a variable, the loaded functions are dereferenced when all copies of that assign target are descoped or assigned to nil. At that point all of the functions in the loaded module can be freed.

Have I explained this OK? Use luac -l on you PC to see what is going on, trawl the app/lua files, and try some test cases on the ESP 8266.

Please see my Developer FAQ for Q&A on nodeMCU Lua before asking a question here that's already been answered. This and the other official documentation should be your first reference for developing Lua Apps

- Sat Apr 04, 2015 5:10 am #13433 Moin Terry,

Carsten,

I see there being two facets here: one is a detailed technical discussion so that the relevant techies can come to a common understanding, and then we write this up in an accessible format for the apps devs who don't want to have to get into the nodeMCU eLua internals to use. The wiki is probably the best place for this.

Exactly. Tips & tricks, hints how to deal with a low resource device on lua script level.
Howto information about high level customization for the firmware build (enabling/disabling modules)
to get more free resources for the slightly more advanced app devs.
Information for the techies to make the script experience more pleasant by tweaking the inner knobs.

A couple of examples of these sorts of issues. I recall you were wondering about how the stack intersected with the heap in another post (I don't have enough karma yet to search for it). Well lua doesn't really have a stack. Everything is a heap object.

Yes I know. The lua stack is really a clever and performant looking design. Performing tail call optimizations
for lua functions does reduce c stack demands and nice to have and may be worth a note in the tips & tricks section.

But unfortunately I should have been more explicit in my post. I really talk about the C stack.
Unlike e.g. java most of the lua libraries are written in C and so C issues matter.

A (contrived) lua example like the following

Code: Select allfunction unescape (s)
s = string.gsub(s,"%%(%x%x)",
    function(s2) return s2:gsub("(%x%x)",
    function(s3) return s3:gsub("(%x%x)",
    function(h) return string.char(tonumber(h,16)) end)
    end)
    end)
return s
end
str = "abc%26def"
print("\n\n\n\n")
res = pcall(unescape(str))
print (res)


Run on an esp01 with a nodemcu firmware with LUAL_BUFFERSIZE=4*1024 bytes (like latest march build)
will trigger a reboot.
I would expect it to
1. give an error of C-stack exhausted without reboot (I am still not 100% what the stack is allowed to be) because
of the pcall
2. give an error of C-stack exhausted with reboot if no pcall is in place

3. On a nodemcu firmware with smaller buffer size I expect it to just run normally A smaller example I tested does run normally, I didn't tested with above example.

I concentrate on getting a proper error message and avoiding the reboot currently (case 1).
I have the code place where I assume that the stack is going to be exhausted (luaD_call)
but throwing an exception just does the reboot.
So I need some high level exception handler in place and currently I struggle with finding out the call stack
leading to that c function.

The lua gsub function is implemented in str_gsub. The locals of that function use the following stack space for the exdample above:

str_gsub: local 3ffff970 .. 3fffe500
str_gsub: buf 3ffff96c .. 3fffe500 = 5228 bytes
lua_checkstack: stack top=3fff5c50 .. base=3fff5c10
str_gsub: local 3fffe3e0 .. 3fffcf70 - 5232 bytes
str_gsub: buf 3fffe3dc .. 3fffcf70 = 5228 bytes

You see
1. Each call needs 5232 bytes on the stack (mostly a buffer of standard size mentioned above)
2. the gsub calls are "stacked"
3. A segment of the lua stack (top/base) is in a different area.

From wiki/doku.php?id=esp8266_memory_map I guess that 3fffc000 is the lower
limit of the stack and the address space below may used for the heap.

As I said I currently concentrate on getting a proper error message on reboot or trying to avoid the reboot
when call is protected.

I got the impression that memory related behavior of auxlib functions may be different between lua and elua.
lua seems to allocate the buffer on the heap and elua - which forms the basis for nodemcu - uses stack based
buffers. Makes sense to me on an MC because stack allocation is more predictable.
But you have to be careful with buffer size ...

Still someone reading?

Carsten

Because of lua semantics, a function has a known number of instructions, constants, locals, up values, which is defined in the Proto block for the function. When your code issues a call, the VM just allocs the correctly sized raw block from the heap to hold all of this, and when you return, this is freed. So there isn't a conventional stack and SP.

I also mentioned this issue of freeing up code. Yes, this does occur when functions are properly dereferenced, but this doesn't happen when you might think it does. Consider the example

Code: Select alllocal x =function()  ... end
...
x=nil

Setting x to nil doesn't free up anything. The assignment binds a closure to a local value, but the enclosure is linked to a hidden function reference in the defining function. You need to free up the latter, to free up the functions. In practice this means that when you assign a load file, etc., to a variable, the loaded functions are dereferenced when all copies of that assign target are descoped or assigned to nil. At that point all of the functions in the loaded module can be freed.

Have I explained this OK? Use luac -l on you PC to see what is going on, trawl the app/lua files, and try some test cases on the ESP 8266.

Get your modules here:

https://www.banggood.com/custlink/KvGGhGF4wG