ESP8266 Community Forum

- Tue Aug 18, 2020 10:46 pm #88365 I am working on a project that involves 100+ Wemos D1 Mini Pro controllers, spread across a warehouse. Each Wemos is acting as a monitor/logger, logging some environmental data to its onboard SPIFFS every 5 seconds, which is working fine. I need to be able to download the log files from each Wemos, both individually and as a group, which means they need to be on the same WiFinetwork.

So to test this on a small scale, I setup 3 Wemos boards which were happily logging away. I have a mini PC running a simple PHP webserver which I plug into my wifi router. When each Wemos is turned on, it connects to the WiFi and then sends a request to the webserver (it has a fixed IP) to register itself with the server. The webserver stores its IP address. I can then connect to the webserver over WiFi from any browser (on the LAN, not over the internet) and the webserver displays a page showing all Wemos controllers and their IPs. I can click on a "get log" button next to any controller to send a request to a /getlog endpoint on the controller. I can also tick checkboxes next to multiple controllers on the page to send the "get log" request to all of them at once. The sketch returns the log file like this:

Code: Select all    ESP8266WebServer server(80);
...
...
    File f = SPIFFS.open("/log.txt", "r");
    server.sendHeader("Content-Type", "text/text");
    server.sendHeader("Content-Disposition", "attachment; filename=log.txt");
    server.sendHeader("Connection", "close");
    server.streamFile(f, "application/octet-stream");
    f.close();

So first question: The communication from webserver to the Wemos controllers over WiFi is currently using curl commands... the html page is using JavaScript to post to a PHP script which builds curl commands for each wemos IP to send them a request over the network... while it works on this small scale (the browser downloads the log file) it feels like an unreliable hack and Im hoping someone has a better idea.

JavaScript:

Code: Select all$.post("getLog.php", {controller: controllerIP}, function(theResponse) {   
});

PHP:

Code: Select all$curlcmd = "curl " . $_POST["controller"] . "/logs";
passthru($curlcmd, $error);
exit();

Second question: Is this a reliable way to return the log file from SPIFFS over WiFi to the webserver, which in turn returns it to the browser as a file download? Is there a more reliable way? The file will grow over time, up to maybe 10Mb in size, so I'm worried about reliably getting the log file from the Wemos to the browser over WiFi. I'll also need to automate this so the webserver proactively polls each controller for its log file every minute maybe, as in case a controller dies I need to ensure I don't lose its onboard log. It would be nice if the controllers just sent their last 1 min worth of data, rather than the full file. Anyone know how to do that? i.e. return chunks of data from a SPIFFS txt file?

Third question: As there can be 100+ of these controllers on the WiFi network, is the above setup the best way to tackle this problem at scale? I was planning on have a set of Ubiquiti AP's chained around the warehouse to ensure each Wemos can connect to the WiFi network.

All input, opinions, feedback and ideas very much welcome. Thanks.

- Wed Sep 09, 2020 4:23 pm #88730 Just a few ideas. Why not keep the data in RAM until you have successfully sent it then put it into SPIFFS. With a few pointers you can keep track of what was sent successfully.

Have you considered using a rechargeable battery backup for the Wemos controllers. If you get a power failure you can detect it and keep on working for a period of time. When the battery starts getting low you can store the status in "EEPROM" and constantly check for good power or shut down. It can also send an error message indicating there is a problem.

I think you can put about 200+ on the network. you would pick preferably a non routable IP (this keeps it off the internet) . then in the last octet change the number for each unit. ie. 123.456.789.??? Appropriate values of course. To keep from going bonkers use a fixed IP for each unit, the last octet would be your id.

Check this link: https://www.extrahop.com/company/blog/2 ... explained/ It will answer your questions about accurcy and realibility.