ESP8266 Community Forum

- Wed Dec 10, 2014 6:07 am #4479 Hi,
posts/notifications crossed - I posted a reply/code here viewtopic.php?f=19&t=702&p=4467#p4467 before I saw your reply and 'redirection'...

I did a lot of testing on this today and I think by breaking up the data into packets manually, similar to the example zeroday presented here, I have a decent workaround. Not fully implemented yet but my plan is to count the length of each string or line that is being sent and when the counter exceeds a limit of something like 800 bytes I'll stop transmitting and wait for the next conn:send call to post the next chunk of data until all data is sent at which point I'll close the connection.

Hmmm, do you already have a plan how to do the "wait for the next conn:send call" as this call only reaches the (detached) callback-code?

I still would like to know what's happening behind the scenes in terms of when data via conn:send is actually being sent.

Me too

Perhaps a delay everynow and then to allow the buffers to be flushed could suffice. I don't know...

I still hope that zeroday can add a socket:flush() or implement something like socket:send(data,NOW) ....

Anyway, thanks for shedding some light on this. I was getting to the point of dumping Lua yesterday based on this issue and some other memory related problems and also random reboots.

Yes, sometimes there is some frustration arising

OTOH (and off topic), it's probably a matter of expectations - see, we have a ~4$ device that can connect to wifi and can be programmed to read sensors, control other devices and speak to the net - all this in an easy language without messing with strbuf's and malloc's Should we really complain about not beeing able to send files of arbitrary length?
Admittedly, I somehow got addicted to nodemcu, and meanwhile i have proof-of-concepts for serving files, sending mail, uploading files via HTTP, scheduling tasks... - all of which are nearly useless for 'production' due to memory constraints and I think maybe I should better start soldering in order to finally control my lights with the smartphone

Anyway, some things still should be ironed out and I hope the networking functions get attention...
Thomas

- Wed Dec 10, 2014 6:36 am #4482 Yes, posts got crossed indeed.

I get that LUA is rather nice for this device. I can whip up a fairly useful application in a matter of minutes. I too have been building a bunch of "proof of concept" applications and they're so close to doing exactly what I want that I get frustrated that I'm too abstracted from the low level code to understand what's going on when I get issues.

With that said, Zeroday has done a great job with Lua for esp. I think it's quite a way ahead of any other developments. The current level of functionality in terms of features available with lua is great right now as it is - I would like to see a freeze on new features and an improvement on the overall perfromance of LUA; mainly memory utillization and just an overall stabililzation of the core lua engine interface with esp.

I'm trying to build a reliable server that can manage all network related functions including storing and serving web pages which could allow me to then offload any (time critical) low level functions to an external cheap and small MCU when required. Some applications I have in mind can and should be done entriely within the esp and some just cannot - I get that but I need to get stability network side first.

Also, I appreciate that probably a lot of issues are still due to core SDK.

Anway, back to the original issue... I've got the workaround working quite well for now. As a side effect I seem to have reclaimed some heap while inside the deepest level function call. Maybe garbage collector is kicking in between packets or something.

Here's what I did in the conn:sent function:

Code: Select allconn:on("sent", function(conn)
         if stillsendingpackets then
            while(packetlength < 1000) do
               local line = file.readline()
               if line then
                  -- Replace variables with values from database motorDB
                  line = line:gsub('<%?lua (.-) %?>', function(code)
                        if not motorDB[code] then motorDB[code]="?" end
                        return(motorDB[code].."  "..node.heap().."  ")
                     end)
                  conn:send(line)
                  packetlength = packetlength + #line
               else
                  file.close()
                  stillsendingpackets = false
                  break
               end
            end
            packetlength = 0
         end
         if not stillsendingpackets then
            conn:close()
            conn = nil   
            stillsendingpackets = nil
         end
      end)


The stillsendingpackets is there to also handle closing of connection of a 'page not found' or 'bad request' message where there is no data to be served from a local file.

- Wed Dec 10, 2014 6:41 am #4483 Inside the conn:receive function after processing header filename etc, heres what I do:

Code: Select all........some conn:receive code....................      
-- Serve webpage
            if file.open(filename, "r") then
               conn:send("HTTP/1.1 200 OK\r\nContent-type: text/html\r\nServer: esp8266\r\nConnection: close\r\n\r\n")   
               packetlength = 0
               stillsendingpackets = true
            else
               conn:send("HTTP/1.1 404 Not Found\r\nContent-type: text/html\r\nServer: esp8266\r\nConnection: close\r\n\r\n")
               conn:send("Page not found")
            end
            filename = nil
         else   
            -- Bad request, send 400 error   
            conn:send("HTTP/1.1 400 Bad Request\r\nContent-type: text/html\r\nServer: esp8266\r\nConnection: close\r\n\r\n")
            conn:send("400 Invalid Request")
....rest of conn:receive code


- Wed Dec 10, 2014 8:01 am #4489

With that said, Zeroday has done a great job with Lua for esp. I think it's quite a way ahead of any other developments. The current level of functionality in terms of features available with lua is great right now as it is - I would like to see a freeze on new features and an improvement on the overall perfromance of LUA; mainly memory utillization and just an overall stabililzation of the core lua engine interface with esp.

right you are and I also agree on the need for network stability...

Code looks good!
Personally, i would go without the 'packetlength'-loop (at the expense of creating many small network packets) in order to:
a) avoid the edge-case of two consecutive long (but < 1000bytes) lines
b) avoid having to wrap my brain around "while sending 3 lines from there - can the second one trigger the actual send and thus fire an additional 'sent' event that occurs before the third one is sent"

...but that's more "better-safe-than-sorry" than a real concern

Thomas