ESP8266 Community Forum

- Wed Nov 25, 2015 5:36 pm #35005 hello,

I am using the following function to grab the contents of a web page but would like to make it get rid of all the header data before returning. Dose any one have a simple way of doing this? I only am trying to get the contents of the page it self. None of the other junk.

Code: Select allString FetchWebUrl(String URLtoGet)
{
  String str;
  String ServerToConnectTo;
  String PageToGet;
  ServerToConnectTo = URLtoGet.substring(0, URLtoGet.indexOf("/"));
  PageToGet = URLtoGet.substring(URLtoGet.indexOf("/"));

  Serial.println(ServerToConnectTo);
  Serial.println(PageToGet);

  WiFiClient client;
  if (client.connect(ServerToConnectTo.c_str() , 80))
  {
    client.print(String("GET " + PageToGet + " HTTP/1.1\r\nHost: " + ServerToConnectTo + "\r\n\r\n"));
    delay(100);
    str = "";
    while (client.available())
    {
      str += String((const char)client.read());
      delay(0);
    }
    return str;
  }
  return "";
}

Serial terminal in the chrome browser see https://serialTerminal.com

http://esp8266basic.com
A BASIC interpreter for your ESP

Working now on Autodrop3d. A 3d printer with automatic part ejection system. https://autodrop3d.com

- Wed Nov 25, 2015 5:54 pm #35007 Since the header and the body of a response are separated by 2 consecutive CRs (or CRLFs), you can simply ignore the header until you found this separation sequence.

- Wed Nov 25, 2015 8:50 pm #35016 I updated my code.

I really hate reinventing the wheel. Dose any one have library or some thing to do this automatically.

Code: Select allString FetchWebUrl(String URLtoGet)
{
  String str;
  String ServerToConnectTo;
  String PageToGet;
  ServerToConnectTo = URLtoGet.substring(0, URLtoGet.indexOf("/"));
  PageToGet = URLtoGet.substring(URLtoGet.indexOf("/"));

  Serial.println(ServerToConnectTo);
  Serial.println(PageToGet);

  WiFiClient client;
  if (client.connect(ServerToConnectTo.c_str() , 80))
  {
    client.print(String("GET " + PageToGet + " HTTP/1.1\r\nHost: " + ServerToConnectTo + "\r\n\r\n"));
    delay(100);
    str = "";
    while (client.available())
    {
      str += String((const char)client.read());
      delay(0);
    }
    Serial.println(str);
    float junkfortest = str.indexOf(String(String(char(13)) + String(char(10)) + String(char(13)) + String(char(10)))      );
    Serial.println("Charactes at");
    Serial.println(junkfortest);
    str = str.substring(junkfortest);
    junkfortest = str.indexOf(String(String(char(10))+"0" )  );
    str = str.substring(0, junkfortest);
    return str ;
  }
  return "";
}

Serial terminal in the chrome browser see https://serialTerminal.com

http://esp8266basic.com
A BASIC interpreter for your ESP

Working now on Autodrop3d. A 3d printer with automatic part ejection system. https://autodrop3d.com

- Thu Nov 26, 2015 10:05 pm #35111 So this function is causing me to have some problems. It soaks up heap like nobody's business.

Dose any one out there have a better way of donig this?

Serial terminal in the chrome browser see https://serialTerminal.com

http://esp8266basic.com
A BASIC interpreter for your ESP

Working now on Autodrop3d. A 3d printer with automatic part ejection system. https://autodrop3d.com