http proxy using EPS8266

Your new topic does not fit any of the above??? Check first. Then post here. Thanks.

Moderator: igrr

 Reply

User avatar

Re: http proxy using EPS8266

By martinayotte
- Thu May 26, 2016 7:59 am #48050 According to your screencast, when it is working, the html body seems to be longer then when it is not working.
Maybe for some reason, the connection is aborting. You could try to print the html into the Serial to figure out what is returned when not working.
User avatar

Re: http proxy using EPS8266

By Vicne
- Tue May 31, 2016 5:30 pm #48352 Hi, martin, and thanks for your suggestion - should have thought of it ;-)

I just changed the code to dump the the contents of each buffer read as you suggested. Here is the new loop:
Code: Select all  while (client.available()) {
    int bytesRead = client.readBytes(buffer, sizeof(buffer));
    Serial.println();
    Serial.println("==========");
    Serial.print(bytesRead);
    Serial.println(" bytes read:");
    Serial.println("==========");
    Serial.write((const char*)buffer, bytesRead);
    browserClient.write((const char*)buffer, bytesRead);
  }
  Serial.println();
  Serial.println("Response sent");


The result is that the beginning of the page is read and returned but then it stops in the middle (the full dump is below). It's strange that it doesn't stop on a buffer boundary, and it always stops at the same position, (except when their nginx proxy replies, which causes a different web page to be returned).
I also tried disabling the last line of the loop (which returns the buffer back to the browser) so that the webpage is only dumped on serial, but it stops at the exact same place.
Looks like the connection to the server is interrupted somehow, but I really don't understand why...

Any idea ?

Thanks.

Vicne

PS: Dump:
Code: Select allServing request
Request is GET / HTTP/1.1
Connecting to remote server
connected
Request sent
Client available

==========
1024 bytes read:
==========
HTTP/1.1 200 OK
Server: Apache
Vary: Host
X-App-Name: homepage
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Date: Tue, 31 May 2016 21:57:09 GMT
Age: 1719
X-API-Version: 5-5
X-Cache: hit
X-PageType: homepage
Connection: close
X-Frame-Options: DENY
Set-Cookie: RMID=007f010154a0574e08b50007;Path=/; Domain=.nytimes.com;Expires=Wed, 31 May 2017 21:57:09 UTC;NYT-wpAB=0089|0;Path=/; Domain=.nytimes.com;Expires=Wed, 01 Jun 2016 21:57:09 UTC
X-Exp-H: 0

002c4c
<!DOCTYPE html>
<!--[if (gt IE 9)|!(IE)]> <!--> <html lang="en" class="no-js edition-domestic app-homepage"  itemscope xmlns:og="http://opengraphprotocol.org/schema/"> <!--<![endif]-->
<!--[if IE 9]> <html lang="en" class="no-js ie9 lt-ie10 edition-domestic app-homepage" xmlns:og="http://opengraphprotocol.org/schema/"> <![endif]-->
<!--[if IE 8]> <html lang="en" class="no-js ie8 lt-ie10 lt-ie9 edition-domestic app-homepage" xmlns:og="http://opengraphprotocol.org/schema/"> <![endif]-->
<!--[if (lt IE
==========
1024 bytes read:
==========
8)]> <html lang="en" class="no-js lt-ie10 lt-ie9 lt-ie8 edition-domestic app-homepage" xmlns:og="http://opengraphprotocol.org/schema/"> <![endif]-->
<head>
    <title>The New York Times - Breaking News, World News & Multimedia</title>
        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" /><script type="text/javascript">window.NREUM||(NREUM={}),__nr_require=function(e,t,n){function r(n){if(!t[n]){var o=t[n]={exports:{}};e[n][0].call(o.exports,function(t){var o=e[n][1][t];return r(o||t)},o,o.exports)}return t[n].exports}if("function"==typeof __nr_require)return __nr_require;for(var o=0;o<n.length;o++)r(n[o]);return r}({1:[function(e,t,n){function r(e,t){return function(){o(e,[(new Date).getTime()].concat(a(arguments)),null,t)}}var o=e("handle"),i=e(2),a=e(3);"undefined"==typeof window.newrelic&&(newrelic=NREUM);var u=["setPageViewName","addPageAction","setCustomAttribute","finished","addToTrace","inlineHit"],c=["addPageAction"],f="api-";i(u,function(e,t){newrelic[t]=r(f+t,"api")}),i(c,function(
==========
1024 bytes read:
==========
e,t){newrelic[t]=r(f+t)}),t.exports=newrelic,newrelic.noticeError=function(e){"string"==typeof e&&(e=new Error(e)),o("err",[e,(new Date).getTime()])}},{}],2:[function(e,t,n){function r(e,t){var n=[],r="",i=0;for(r in e)o.call(e,r)&&(n[i]=t(r,e[r]),i+=1);return n}var o=Object.prototype.hasOwnProperty;t.exports=r},{}],3:[function(e,t,n){function r(e,t,n){t||(t=0),"undefined"==typeof n&&(n=e?e.length:0);for(var r=-1,o=n-t||0,i=Array(0>o?0:o);++r<o;)i[r]=e[t+r];return i}t.exports=r},{}],ee:[function(e,t,n){function r(){}function o(e){function t(e){return e&&e instanceof r?e:e?u(e,a,i):i()}function n(n,r,o){e&&e(n,r,o);for(var i=t(o),a=l(n),u=a.length,c=0;u>c;c++)a[c].apply(i,r);var s=f[g[n]];return s&&s.push([m,n,r,i]),i}function p(e,t){w[e]=l(e).concat(t)}function l(e){return w[e]||[]}function d(e){return s[e]=s[e]||o(n)}function v(e,t){c(e,function(e,n){t=t||"feature",g[n]=t,t in f||(f[t]=[])})}var w={},g={},m={on:p,emit:n,get:d,listeners:l,context:t,buffer:v};return m}function i(){return new r}var a="nr@contex
==========
1024 bytes read:
==========
t",u=e("gos"),c=e(2),f={},s={},p=t.exports=o();p.backlog=f},{}],gos:[function(e,t,n){function r(e,t,n){if(o.call(e,t))return e[t];var r=n();if(Object.defineProperty&&Object.keys)try{return Object.defineProperty(e,t,{value:r,writable:!0,enumerable:!1}),r}catch(i){}return e[t]=r,r}var o=Object.prototype.hasOwnProperty;t.exports=r},{}],handle:[function(e,t,n){function r(e,t,n,r){o.buffer([e],r),o.emit(e,t,n)}var o=e("ee").get("handle");t.exports=r,r.ee=o},{}],id:[function(e,t,n){function r(e){var t=typeof e;return!e||"object"!==t&&"function"!==t?-1:e===window?0:a(e,i,function(){return o++})}var o=1,i="nr@id",a=e("gos");t.exports=r},{}],loader:[function(e,t,n){function r(){if(!w++){var e=v.info=NREUM.info,t=s.getElementsByTagName("script")[0];if(e&&e.licenseKey&&e.applicationID&&t){c(l,function(t,n){e[t]||(e[t]=n)});var n="https"===p.split(":")[0]||e.sslForHttp;v.proto=n?"https://":"http://",u("mark",["onload",a()],null,"api");var r=s.createElement("script");r.src=v.proto+e.agent,t.parentNode.insertBefore(r,t)}}}
==========
1024 bytes read:
==========
function o(){"complete"===s.readyState&&i()}function i(){u("mark",["domContent",a()],null,"api")}function a(){return(new Date).getTime()}var u=e("handle"),c=e(2),f=window,s=f.document;NREUM.o={ST:setTimeout,CT:clearTimeout,XHR:f.XMLHttpRequest,REQ:f.Request,EV:f.Event,PR:f.Promise,MO:f.MutationObserver},e(1);var p=""+location,l={beacon:"bam.nr-data.net",errorBeacon:"bam.nr-data.net",agent:"js-agent.newrelic.com/nr-918.min.js"},d=window.XMLHttpRequest&&XMLHttpRequest.prototype&&XMLHttpRequest.prototype.addEventListener&&!/CriOS/.test(navigator.userAgent),v=t.exports={offset:a(),origin:p,features:{},xhrWrappable:d};s.addEventListener?(s.addEventListener("DOMContentLoaded",i,!1),f.addEventListener("load",r,!1)):(s.attachEvent("onreadystatechange",o),f.attachEvent("onload",r)),u("mark",["firstbyte",a()],null,"api");var w=0},{}]},{},["loader"]);</script>
<link rel="shortcut icon" href="https://static01.nyt.com/favicon.ico" />
<link rel="apple-touch-icon-precomposed" sizes="144×144" href="https://static01.nyt.com/
==========
720 bytes read:
==========
images/icons/ios-ipad-144x144.png" />
<link rel="apple-touch-icon-precomposed" sizes="114×114" href="https://static01.nyt.com/images/icons/ios-iphone-114x144.png" />
<link rel="apple-touch-icon-precomposed" href="https://static01.nyt.com/images/icons/ios-default-homescreen-57x57.png" />
<meta name="sourceApp" content="nyt-v5" />
<meta id="applicationName" name="applicationName" content="homepage" />
<meta id="foundation-build-id" name="foundation-build-id" content="" />
<link rel="canonical" href="http://www.nytimes.com" />
<link rel="alternate" type="application/rss+xml" title="RSS" href="http://www.nytimes.com/services/xml/rss/nyt/HomePage.xml" />
<link rel="alternate" media="only screen and (max-width: 640pResponse sent
BrowserClient served
User avatar

Re: http proxy using EPS8266

By martinayotte
- Wed Jun 01, 2016 3:16 pm #48414 I've tried with my own piece of code :

Code: Select all  WiFiClient client;
  if (!client.connect("www.nytimes.com", 80)) {
    Serial.println("Failed to connect with 'google.com' !");
  }
  else {
    int timeout = millis() + 5000;
    client.print("GET / HTTP/1.1\r\nHost: www.nytimes.com\r\nConnection: close\r\n\r\n");
    while (client.available() == 0) {
      if (timeout - millis() < 0) {
        Serial.println(">>> Client Timeout !");
        client.stop();
        return;
      }
    }
    int size;
    while ((size = client.available()) > 0) {
      uint8_t* msg = (uint8_t*)malloc(size);
      size = client.read(msg, size);
      Serial.write(msg, size);
      free(msg);
    }
  } 


... and I see strange behavior with this NYTimes server ... :?
There is latencies during the response, like a delay between packets, and effectively it stop before the real content end.

I have no idea what going on ... :(
Probably a bug in LWIP of ArduinoESP ...

EDIT : I've posted this description into the gitter chat ...
User avatar

Re: http proxy using EPS8266

By Me-no-dev
- Wed Jun 01, 2016 5:04 pm #48418 If you want to proxy data as it comes, you need to go async on both the server and the client. Links for relevant libraries are in my signature :)
Data in async is handled packet by packet, so you only transfer 1460 bytes at a time.
 Reply
Return to General Discussions