ESP8266 Community Forum

- Sun Sep 19, 2021 9:34 am #92377 I have made some progress in bringing "external" LWIP to V2 (and still working...). It is looking good now.

The main hurdles are:
- unclear espressif code inside LWIP that needs to remain for LWIP to work with ESP
- unclear espressif code inside LWIP that does not need to remain (it seems they contain fixes for older LWIP versions or their own code in the SDK)
- the branch used by Espressif has no "git relation" to the LWIP V2 branches. Normally you'd keep the espressif code on top and keep rebasing all the way through the last commit, fixing conflicts here and there. But there is no path to do so.

So here is what I did:
- extracted every single piece of code added by espressif (which I luckily already did for the previous version)
- checked what of that code was really necessary (if all UDP/TCP/ICMP/RAW still works after removing it, remove it permanently), so the changes towards the upstream LWIP would be as small as possible (less chance of conflicts)
- rewrote some espressif code, so it has less impact on git (smaller change sets)
- took a commit in the "V2" branch of lwip that has it's tree look most like the "espressif" base branch.
- reworked all commits/patches that insert Espressif code to "fit" there
- fingers crossed

It didn't work at first. It appears that LWIP had inserted a field in the "netif" struct that the SDK lib doesn't know about (all structs MUST remain binary compatible because we can't recompile the SDK). Fortunately it was a field that's only necessary for IPv6, and it was incorrectly inserted because IPv6 isn't enabled (yet). That means when I will start working on IPv6, this field will have to move outside the area that the SDK code uses.

So now it works with the "LWIP V2" branch and I am gradually bringing it to "now" by rebasing it, keeping the espressif (and my) code on top. At some time I hope I will have read HEAD and we will be in sync with LWIP V2.

Note: I am not using all features of lwip or the SDK. So there is always a chance something stops working. But at least these are working (and tested by me):
- TCP
- UDP
- ping (RAW)
- DHCP client
- DHCP server
- access point mode

I'm here now, so I guess still some work to go:

Code: Select allcommit 7c9b545b301ee11f73b6fb123d811079c5137865
Author: Simon Goldschmidt <goldsimon@gmx.de>
Date:   7 years ago

    Fixed CHANGELOG to correctly show the differences to 1.4.1 (got messed up because git master was developed in parallel to the 1.4.1 branch, both starting at 1.4.0)


But as long are there are no conflicts (9 out of 10 rebase actions), it all goes pretty quickly.

- Sun Sep 19, 2021 7:32 pm #92381 AMAZING work mate!!! What a huge amount of effort!

Regarding the extra field - is it possible to move it to the end of the structure or create a union with the v1.x/Espressif version so that you can use the v2 structures in LWIP under the hood but pass a v1.x compatible structure back and forth?

Or is the problem more around the calls made to LWIP from the SDK code only passing in a v1.x structure?

- Mon Sep 20, 2021 2:56 am #92386 At the moment I think it's indeed very well possible to move the field to the end of the struct. I will certainly do that when/if I get to the point where I can enable ipv6 (together with other added members). I changed the #ifdef in this case though, because the member shouldn't be there anyway, if ipv6 is disabled.

I already have had to roll back parts of LWIP commits that broke binary compatibility. I am curious how long I will get away with that, if I can reach the HEAD commit and all still working... Fingers crossed! Still a few thousand of commits to go, so... (I am guessing I am at 1/3 of all now). Most of the commits are about PPP and ipv6 though, so won't "hurt" us.

The pity is that Espressif could have done this work easily, and in fact, actually did, but just for the RTOS SDK .

It's a bit of a pity I don't see improvement in tcp performance (yet). I have this issue where once in a while TCP segments remain unacked for eternity, the only solution is to drop the data and re-send it from the application layer. It's not a network-layer thing, as I don't have such issues with UDP, never.

- Mon Sep 20, 2021 6:53 pm #92390 Thank you for all the work you're doing - this is incredible. I still have some hope that maybe one day Espressif will bring in community devs like you (me) to maintain the NonOS SDK. This is what seems to have happened with the RTOS release because there are a lot of commiters that aren't Espressif employees