Http download speed

Your new topic does not fit any of the above??? Check first. Then post here. Thanks.

Moderator: igrr

 Reply

User avatar

Http download speed

By Vicne
- Sat Aug 13, 2016 6:22 pm #52879 Hi,

I am trying to understand where the bottleneck is when browsing files from the SPIFFS, so I thought I would test separately the SPIFFS read speed and the HTTP download speed. Thinking the latter would probably not be the culprit, I wanted to rule it out so I started by wrapping up a sketch that allows me to download a variable size "file" (generated on the fly). The full sketch is below.

What I'm observing is that building the response one character at a time by repeatedly calling client.write() causes a very high overhead, slowing data transfer.
I compared with a static response using server.send() and there is a factor 1000 in transfer speed for a 1KB payload.
I read that the Nagle algorithm could be in the way so I added a "client.setNoDelay(true);", to no avail.

See this screencast for a how it feels.

So my questions are: what is the reason for this slow transfer and how can I reliably test http download speed ?

And while you're at it, if you have any experience on transfer speed or SPIFFS speed and how to optimize them, please feel free to comment of course.

Kind regards,

Vicne

PS: Test sketch:
Code: Select all#include <Arduino.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include <WiFiManager.h>          // from lib https://github.com/tzapu/WiFiManager

ESP8266WebServer server(80);

void generateFixedContent() {
  server.send(200, "application/octet-stream",
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789"
    );
}

void generateVariableContent() {
  if(server.args() == 0) return server.send(500, "text/plain", "Parameter 'size=xxx' is required");
 
  int length = atoi(server.arg(0).c_str());
  if(length <= 0) return server.send(500, "text/plain", "Wrong size for parameter");
 
  WiFiClient client = server.client();

  client.setNoDelay(true);
 
  String header = "HTTP/1.0 200 OK\r\n";
  header += "Content-Type: application/octet-stream\r\n";
  header += "Content-Length: " + String(length) + "\r\n";
  header += "Content-Disposition: attachment; filename=data_" + String(length) + ".dat\r\n";
  header += "Connection: close\r\n";
  header += "\r\n";
  client.write(header.c_str(), header.length());
  for (int i = 0; i< length; i++) {
    client.write("A", 1);
  }
}


void setup() {
  WiFiManager wifiManager;
  wifiManager.autoConnect("SpeedTest_CFG");

  server.on("/", [](){
    server.send(200, "text/html", "<html><body>Use <a href='/fixed'>/fixed</a> or <a href='/variable?size=1000'>/variable?size=1000</a> to generate 1000 bytes</body></html>");
  });

  server.on("/fixed", generateFixedContent);
 
  server.on("/variable", generateVariableContent);

  server.onNotFound([](){     
      server.send(404, "text/plain", "FileNotFound");
  });

  server.begin();
}


void loop() {
  server.handleClient();
}
User avatar

Re: Http download speed

By martinayotte
- Sun Aug 14, 2016 11:31 am #52938 The fact that you are sending 1 character at a time with client.write("A", 1) is the bottleneck, because there will be one packet per character.
You should accumulate characters in a buffer, usually the length of a MTU, so 1460 bytes, and then transmit the chunk in a single client.write(buf, sizeof(buf)).
User avatar

Re: Http download speed

By Vicne
- Sun Aug 14, 2016 3:42 pm #52951 Thanks @martinayotte,

OK, got it.

So it would mean the ESP sends around 20 packets/sec max and if they are full, it would mean the maximum bitrate would be 20x1460=29kB/s or 228kbps ? Sounds a bit low...

I will test using a 1460bytes buffer when I have time and report back.


Vicne
User avatar

Re: Http download speed

By Vicne
- Tue Aug 16, 2016 3:13 pm #53102 Hi, all,

I made further tests and I confirm that the speed observed is between 100 and 300 kbps with payloads around 100KB, which in my experience makes it unsuitable for a classic "2016" webpage (say 300 KB including a few JS libs and images, which means around 10 seconds).
I am a bit surprised by that speed as even 802.11b is already rated between 1 and 11Mbps.
I noticed that for very small amounts of data (around 1KB roughly), the speed can be doubled by combining header and content in the same package. Indeed ESP8266WebServer.send() currently calls:
Code: Select allsendContent(header);
sendContent(content);


which creates two separate packets.

At the expense of a bit of RAM, it can be rewritten as follows and catch the low-hanging fruits:
Code: Select all   if (header.length() + content.length() < HTTP_DOWNLOAD_UNIT_SIZE) {
      sendContent(header + content);
   }
   else {
      sendContent(header);
      sendContent(content);
   }


However, the influence of this extra header packet becomes negligible with bigger payloads.
I dug the code down to the lwip stack and could not find an obvious reason.
Is the speed limited by the ESP's hardware or software ?
Could anyone achieve higher speeds or is this close to the maximum of the chip ?

Kind regards,

Vicne
 Reply
Return to General Discussions