ESP8266 Community Forum

- Thu May 10, 2018 5:03 pm #75865 I have a file on a HTTPS server. I can get the URL I send into DownloadMenuFile() [<see below] and put it into a browser or into Postman and get the file but when I use my code below, the GET returns 200, the file size is correct and the stream pointer is valid but the stream size is 0. What is going on?

The URLs are one use because of the token and look like this, names changed to protect the guilty...
https://someplace.cloud/app/upload/devi ... 927e32dcb8
The URL above is passed into the function below.
The Fingerprint is valid. If I change it the connection is not established.

// Genuno Settings
// Board: NodeMCU 1.0 (ESP-12E Module)
// ESP8266Libraries -> ESP8266-2.4.0
// Flash Mode: DIO
// Flash Freq: 40MHz
// CPU Freq: 80MHz
// Flash Size: 4M(3M SPIFFS)
// Reset Method: ck
// Upload Speed: 115200
// Programmer: AVRISP mkii

Code: Select allvoid DownloadMenuFile(char downUrl[])
{
  HTTPClient httpClient;
  char buf[130] = {0};
  uint8_t bufferD[128] = {0}; 
  int httpCode = 0;
   
  // We now create a URI for the request
  DEBUG_PRINT(downUrl);
  httpClient.begin(downUrl, "73 3A E4 C2 BA 9F B8 FC D9 BD 6D 1C 64 7F FA 6B B8 48 F4 08");

  testStack(); // Plenty of stack
   
  httpClient.setReuse(true);
 
  httpCode = httpClient.GET();
  Serial.println(httpCode); // I get 200

  if (httpCode >0)
  {
 
    DEBUG_PRINT("downloadFile Connected");
   
    int len = httpClient.getSize();
    Serial.print("File size: ");
    sprintf(buf, "%d", len); // The length is valid
    Serial.println(buf);   
    WiFiClient* myStream = httpClient.getStreamPtr();
       
    if(myStream ) // This is always valid
    {
      while (httpClient.connected() && (len > 0 || len == -1))
      {
        // get available data size
        size_t sizeStream = myStream->available();
        Serial.print("Stream Size:");
        sprintf(buf,"%d",sizeStream);
        Serial.println(buf);     // THIS IS ALWAYS 0     
           
        if (sizeStream) // I NEVER GET IN HERE
        {
          // read up to buffer size
          int c = myStream->readBytes(bufferD, ((sizeStream > sizeof(bufferD)) ? sizeof(bufferD) : sizeStream));         
   
          sprintf(buf, "Data: %s", bufferD);
          Serial.println(buf);
          Serial.print("bytes read:");
          Serial.println(c);
   
          if (len > 0)
          {
            len -= c;
          }
        }
        else
          len = -2;
        delay(1);
      }
    }     
  }
  else
  {
    DEBUG_PRINT("ERROR: downloading file from cloud.");   
    handleHTTPError(httpCode);   
  }
  httpClient.end();   //Close connection

  DEBUG_PRINT("Complete");
 
  return ;
}


Thanks in advance...

- Fri May 11, 2018 8:08 am #75876 I get the feeling that the way the data is being sent to me is not working with the stream.
What could that be? I am not familiar with the different ways data can be sent over HTTP.

Can I post the header I get from PostMan when I get the file and someone can tell me?

What ever happened to:
WiFiClient client
...
if (client.available()) {
char c = client.read();
Serial.print(c);
}

- Fri Jun 15, 2018 7:38 am #76454 So the problem was that as soon as I call any function on the stream object like stream->available() the behind the scenes code tries to allocate a buffer for the entire size of the payload of the GET call. The code was failing to allocate that buffer and not letting me know. I just got a return available amount of 0. When I turned on the dubug messages, which were off because they take even more memory that I did not have, it was clear what was happening. This kinds of defeats the purpose of stream->readBytes since you have to have enough memory for the entire payload just so I can read it in chunks from internal memory somewhere.

So the solution was to ask for the file in chunks from the server by adding the header "Range" to the GET and collecting the Content-Range return header so I could know the size of the file and where I was in the download process.

Code: Select allDEBUG_PRINT(downUrl);
httpClient.begin(downUrl, "73 3A E4 ... 08"); 
httpClient.addHeader("Range","bytes=0-255");
const char* headerNames[] = { "Content-Range"};
httpClient.collectHeaders(headerNames,1);
httpClient.setReuse(true);
httpCode = httpClient.GET(); 


Then parse the file size out of the first returned Content-Range header. Get the payload out of the first packet and write it to file as above then make more GET calls with the Range header incremented to get all the packets:

Code: Select allhttpClient.end();   // End this HTTP call because we read all the data
delay(2);   

// Configure a new HTTP GET
sprintf(rangeHeader,"bytes=%d-%d",rangeStart,rangeStart+bufferSize-1);
httpClient.addHeader("Range",rangeHeader);
rangeStart = rangeStart+bufferSize;
httpCode = httpClient.GET();
stream = httpClient.getStreamPtr(); // Don't forget to get a new stream pointer


I hope this helps someone.