ESP8266 Community Forum

- Sun May 22, 2016 3:56 pm #47846 Hi,

My project requires that a browser controls the ESP8266 via wifi while using data from a public website. Unfortunately, accessing two different domains that way is not possible due to cross-site scripting limitations in browsers.

So I thought the best solution could be to have the ESP8266 proxy the requests to the public website, so that the browser only "talks" to the ESP8266.

I succeeded in having the browser request data from the ESP, which itself requests data from the external website, receives the response and forwards it to the browser.

The problem is that this only works with relatively small responses due to memory constraints on the ESP. So I would like a kind of "streaming" solution that forwards the response to the browser while it is still being read from the server.

Any idea how to achieve that streaming ?

Kind regards,

Vicne

- Mon May 23, 2016 9:16 pm #47910 Have a look at the Stream library.

Use an array of bytes (called a buffer) and use a loop to continually read into the buffer, readBytes(), then write the contents to your browser connection, until there's no more bytes available. Good luck!

- Tue May 24, 2016 7:47 am #47929 WiFiClient is already derived from Stream ...

- Wed May 25, 2016 5:12 pm #48019 Hi, davino, martin,

Thanks for your replies

I worked on a version using streams and I think I got it almost right, but the result is not perfect yet.
I'm attaching the code below, and to make it simpler, I just tried to proxy the New-York Times homepage (just the html, not the images).

Here is a screencast of how it behaves:
http://screencast.com/t/2IKIEhoJVBij
I'm first loading the page directly in Chrome so we have the reference. Then I'm loading the page through the ESP. As you can see, most of the times, I'm getting a blank page of only about 5KB (and I cannot even view its source, which is weird), and only once I could get the full page loaded.

Also, the processing looks extremely slow to me (no matter if it works or not). It looks like we have download bursts interleaved with long pauses.

Here is the code I'm using. Sorry, it's a bit long with all the comments, but it should be easy to use to reproduce:

Code: Select all#include <ESP8266WiFi.h>
 
const char* ssid = "********";
const char* password = "****************";
 
WiFiServer localServer(80);

const char* remoteHost = "www.nytimes.com";

char buffer[1024];
 
void setup() {
  Serial.begin(115200);
  delay(10);
   
  // Connect to WiFi network   
  WiFi.begin(ssid, password);
   
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
   
  // Start the server
  localServer.begin();

  // Print the IP address
  Serial.print("Started. Connect to: http://");
  Serial.print(WiFi.localIP());
   
}
 
void loop() {
  // Check if a client browser has connected
  WiFiClient browserClient = localServer.available();
  if (!browserClient) {
    return;
  }
   
  // Wait until the client browser sends some data
  Serial.println("Serving request");
  while(!browserClient.available()){
    delay(1);
  }

  // Read the first line of the request (and ignore it)
  String request = browserClient.readStringUntil('\r');
  Serial.println("Request is " + request);
  browserClient.flush();
   
  // Connect to the real server
  WiFiClient client;
  Serial.println("Connecting to remote server");
  if (!client.connect(remoteHost, 80)) {
    // Error connecting to real server
    Serial.println("Error contacting remote server");
    return;
  }
 
  Serial.println("connected");
  // Make a HTTP request:
  client.println(String("GET / HTTP/1.1\r\n") +
             "Host: " + remoteHost + "\r\n" +
             "Connection: close\r\n\r\n"); 
  Serial.println("Request sent");
 
  // Wait for response
  while(!client.available()){
    delay(1);
  }
  Serial.println("Client available");

  // And forward it to the browser
  while (client.available()) {
    int bytesRead = client.readBytes(buffer, sizeof(buffer));
    Serial.print(bytesRead);
    Serial.println(" bytes read");
    browserClient.write((const char*)buffer, bytesRead);
  }
  Serial.println("Response sent");
 
  delay(1);
  Serial.println("BrowserClient served");
}
 

Do you see a reason why it behaves that way ?

Best regards,

Vicne