ESP8266 Community Forum

- Thu Jun 02, 2016 6:18 pm #48485 You're welcome !
And good luck with your project, especially that it seems simpler than trying to get the sync issue working like discussed on the chat.

- Sat Jun 04, 2016 5:27 am #48553 Hi, all,

I just tried the "2-step" proxying using a cache on SPIFFS, but to my surprise, I'm encountering the same weird "truncated page" issue.

Here is the code I'm using now:

Code: Select all#include <ESP8266WiFi.h>
#include <FS.h>
 
const char* ssid = "********";
const char* password = "********";
 
WiFiServer localServer(80);

const char* remoteHost = "www.nytimes.com";

long cacheTime;
#define CACHE_TTL 1*60*1000 // 1 minute

uint8_t buffer[1024];
 
void setup() {
  Serial.begin(115200);
  delay(10);
   
  Serial.println();
  Serial.print("Initializing Wi-Fi");
  // Connect to WiFi network   
  WiFi.begin(ssid, password);   
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println(" Done.");
   
  // Start the server
  localServer.begin();

  Serial.println("Initializing SPIFFS cache...");
  if (!SPIFFS.begin()) {
    Serial.println("SPIFFS init failed");
  }
  Dir dir = SPIFFS.openDir("/cache");
  while (dir.next()) {
      Serial.println("Deleting " + dir.fileName() + " from cache...");
      SPIFFS.remove(String("/cache") + dir.fileName());
  }
  cacheTime = 0;

  Serial.print("Ready. Please direct your browser to: http://");
  Serial.println(WiFi.localIP());   
}
 
void loop() {
  // Check if a client browser has connected
  WiFiClient browserClient = localServer.available();
  if (!browserClient) {
    return;
  }
   
  // Wait until the client browser sends some data
  Serial.println("Serving request");
  while(!browserClient.available()){
    delay(1);
  }

  // Read the first line of the request (and ignore it)
  String request = browserClient.readStringUntil('\r');
  Serial.println("Request is " + request);
  browserClient.flush();

  // Test if we can use the cache
  long now = millis();

  if (   cacheTime == 0 /* no cache */
      || cacheTime > now /* millis() wrapped around */
      || now - cacheTime > CACHE_TTL /* cache is outdated */) {
        Serial.println("Filling up cache");

    // Connect to the real server
    WiFiClient client;
    Serial.println("Connecting to remote server");
    if (!client.connect(remoteHost, 80)) {
      // Error connecting to real server
      Serial.println("Error contacting remote server");
      return;
    }
   
    Serial.println("connected");
    // Make a HTTP request:
    client.println(String("GET / HTTP/1.1\r\n") +
               "Host: " + remoteHost + "\r\n" +
               "Connection: close\r\n\r\n"); 
    Serial.println("Request sent");
   
    // Wait for response
    while(!client.available()){
      delay(1);
    }
    Serial.println("Client available");
 
    // Prepare cache file
    File cache = SPIFFS.open(String("/cache/") + remoteHost, "w");
    if (!cache) {
      Serial.println("Could not create cache file");
    }

    while (client.available()) {
      int bytesRead = client.readBytes(buffer, sizeof(buffer));
      cache.write(buffer, bytesRead);
      Serial.print(bytesRead);
      Serial.println(" bytes downloaded to cache");
    }
    cache.close();
    cacheTime = now;
  }

  // And return cached value to the browser
 
  // Open cache file for reading
  File cache = SPIFFS.open(String("/cache/") + remoteHost, "r");
  if (!cache) {
    Serial.println("Could not open cache file");
  }

  // Read it and stream it back
  while (cache.available()) {
    int bytesRead = cache.read((uint8_t *)buffer, sizeof(buffer));
    browserClient.write((uint8_t *)buffer, bytesRead);
    Serial.print(bytesRead);
    Serial.println(" bytes returned to browser from cache");
  }
 
  cache.close();
 
  Serial.println();
  Serial.println("Response sent");
}

And here is the log:

Code: Select allInitializing Wi-Fi............. Done.
Initializing SPIFFS cache...
Ready. Please direct your browser to: http://192.168.0.18
Serving request
Request is GET / HTTP/1.1
Filling up cache
Connecting to remote server
connected
Request sent
Client available
1024 bytes downloaded to cache
1024 bytes downloaded to cache
1024 bytes downloaded to cache
1024 bytes downloaded to cache
1024 bytes downloaded to cache
720 bytes downloaded to cache
1024 bytes returned to browser from cache
1024 bytes returned to browser from cache
1024 bytes returned to browser from cache
1024 bytes returned to browser from cache
1024 bytes returned to browser from cache
720 bytes returned to browser from cache

Response sent

As you can see, I'm again getting that 5x1024+720 bytes truncated response instead of the full 100k+ page.
Note that writing that, I'm just realizing that (5x1024+720)/4 = 1460 , so we stop after exactly 4 ethernet frames...
I first thought about a potential timeout due to the time it takes to return these data to the browser client (first implementation) or to write it to flash (this implementation), but it wouldn't explain why writing to serial (which is much slower) works.
So maybe it's the contrary: we drain the queue of incoming messages so fast that the client.available() becomes false for a short time and then we exit the loop prematurely.
But that wouldn't explain why Serial + browser forwarding failed... Really strange.
I will test some more with delays to see how it behaves...

Apart from that, the cache logic seems to work pretty well, as subsequent calls within the same minute successfully hit the cached copy.

Kind regards,

Vicne

- Sat Jun 04, 2016 9:59 am #48565 Hi Vicne,

The reason that you don't have whole page has been discussed with @igrr on gitter the first day I posted the issue there, it is because sometimes the client.available() returns zero because the server didn't send any thing yet although the connection is still up. @igrr provided another way to workaround that using HTTPClient instead and using http.connected() to figure out if we need to stay in the loop. Check again @igrr code at https://gist.github.com/igrr/a81e10c375 ... c1a18b0eb4
But ! Big But ! Trying to cache into SPIFFS this content, I've faced another issue : I get some WTD exceptions, sometime at the middle of the page, sometimes earlier, sometimes further. The biggest size I got was around 80K.
If I only write chunks size into the SPIFFS instead of the actual content, I got the whole page downloaded (but content thrown away).
So, we are back to SquareOne ...

- Sat Jun 04, 2016 11:28 am #48567 Hi,

The reason that you don't have whole page has been discussed with @igrr on gitter the first day I posted the issue there, it is because sometimes the client.available() returns zero because the server didn't send any thing yet although the connection is still up.

Yes, I figured that out and used another workaround by only stopping download after a 100ms timeout without any data, using:

Code: Select alldo {
      while (client.available()) {
        int bytesRead = client.readBytes(buffer, sizeof(buffer));
        cache.write(buffer, bytesRead);
      }
      delay(100);
    }
    while (client.available());

@igrr provided another way to workaround that using HTTPClient instead and using http.connected() to figure out if we need to stay in the loop. Check again @igrr code at https://gist.github.com/igrr/a81e10c375 ... c1a18b0eb4

That is much cleaner indeed. I'll give it a go.

But ! Big But ! Trying to cache into SPIFFS this content, I've faced another issue : I get some WTD exceptions, sometime at the middle of the page, sometimes earlier, sometimes further. The biggest size I got was around 80K.
If I only write chunks size into the SPIFFS instead of the actual content, I got the whole page downloaded (but content thrown away).
So, we are back to SquareOne ...

Jeez. Really strange... Looks like mixing technologies causes race conditions or something.
On my side, tests now exhibit a completely new behaviour: the start of the page is missing (starts right in the middle of a word). I switched to eetimes.com but I have the same issue. I must have messed with something. Investigating...

Kind regards,

Vicne