fetching a web page but discarding header data
Posted:
Wed Nov 25, 2015 5:36 pmby Mmiscool
hello,
I am using the following function to grab the contents of a web page but would like to make it get rid of all the header data before returning. Dose any one have a simple way of doing this? I only am trying to get the contents of the page it self. None of the other junk.
Code: Select allString FetchWebUrl(String URLtoGet)
{
String str;
String ServerToConnectTo;
String PageToGet;
ServerToConnectTo = URLtoGet.substring(0, URLtoGet.indexOf("/"));
PageToGet = URLtoGet.substring(URLtoGet.indexOf("/"));
Serial.println(ServerToConnectTo);
Serial.println(PageToGet);
WiFiClient client;
if (client.connect(ServerToConnectTo.c_str() , 80))
{
client.print(String("GET " + PageToGet + " HTTP/1.1\r\nHost: " + ServerToConnectTo + "\r\n\r\n"));
delay(100);
str = "";
while (client.available())
{
str += String((const char)client.read());
delay(0);
}
return str;
}
return "";
}
Re: fetching a web page but discarding header data
Posted:
Wed Nov 25, 2015 5:54 pmby martinayotte
Since the header and the body of a response are separated by 2 consecutive CRs (or CRLFs), you can simply ignore the header until you found this separation sequence.
Re: fetching a web page but discarding header data
Posted:
Wed Nov 25, 2015 8:50 pmby Mmiscool
I updated my code.
I really hate reinventing the wheel. Dose any one have library or some thing to do this automatically.
Code: Select allString FetchWebUrl(String URLtoGet)
{
String str;
String ServerToConnectTo;
String PageToGet;
ServerToConnectTo = URLtoGet.substring(0, URLtoGet.indexOf("/"));
PageToGet = URLtoGet.substring(URLtoGet.indexOf("/"));
Serial.println(ServerToConnectTo);
Serial.println(PageToGet);
WiFiClient client;
if (client.connect(ServerToConnectTo.c_str() , 80))
{
client.print(String("GET " + PageToGet + " HTTP/1.1\r\nHost: " + ServerToConnectTo + "\r\n\r\n"));
delay(100);
str = "";
while (client.available())
{
str += String((const char)client.read());
delay(0);
}
Serial.println(str);
float junkfortest = str.indexOf(String(String(char(13)) + String(char(10)) + String(char(13)) + String(char(10))) );
Serial.println("Charactes at");
Serial.println(junkfortest);
str = str.substring(junkfortest);
junkfortest = str.indexOf(String(String(char(10))+"0" ) );
str = str.substring(0, junkfortest);
return str ;
}
return "";
}
Re: fetching a web page but discarding header data
Posted:
Thu Nov 26, 2015 10:05 pmby Mmiscool
So this function is causing me to have some problems. It soaks up heap like nobody's business.
Dose any one out there have a better way of donig this?