ESP8266 Community Forum

I am running some overly abusive tests on my project. The current test in question is a server, serving four WebSockets. I am trying to push data out (small messages < 50 bytes each) as fast as possible. At any rate below say around 3 KB/sec each (12 KB/sec total) the sever and clients will merrily work far longer than I have patience to test... I use 100 hours as a cutoff for a successful test and of my patience.

I've bumped up the send rate to 1000 sends/socket/second. I don't know if its just coincidence, but the the byte rate per socket is hovering right around 2^15 = 32KB/sec. I've run this test a couple of times since it fails in relatively short order. The last one took only 47 minutes to fail. I'm well past trivial things like memory leaks as over 270 MB have been pushed out.

I started searching the Internet about the exception I get and ran across - Ets intr lock nest (#6484)
https://git.riper.fr/ext/esp8266/commit/d8531cb2c4aec39d917fed8d642afe86af59e1c6

I'm not sure what this is supposed to be telling me... Its a little more Advanced than I know.

Here is the Exception Decoder Output. The first line of code that I actually wrote is about 5 lines up from the bottom. What I noted was the article listed above mentioned some arbitrary count of 7 that sounds like they didn't expect this to ever reach it and yet I see three blocks of 7 calls to this function. So, I think I hitting this magic number. I just don't know what that is suppose to be telling me or how I go about fixing it.

Any help is greatly appreciated.

You wrote to memory that's not writeable - possibly you tried to get some memory, which failed and then you didn't check a null pointer return

davydnorris wrote:You wrote to memory that's not writeable - possibly you tried to get some memory, which failed and then you didn't check a null pointer return

I am an Advanced user... just not advanced enough to understand the sited issue #6484.

If you are an Advanced User and sincerely want to help, you'll need to read beyond the first line of the my OP and of the exception. Although your response is technically correct... the author of the code that failed is not me. Unfortunately my code is 54 calls up the stack! I also checked all values in my function. That exact same method call (of mine) has been called over a billion times (literally) with the same values and processed over a 300 GB of data... at a lower rate in a successful 100 hour test. At this higher rate, it now fails... after only millions of calls. The quickest failure took 45 minutes. This last one, took nearly four hours and had processed almost 2GB before giving the same exception.

So, at the moment, I am asking for advice about how I, at 54 calls up the stack in my routine, can compensate for the failing of what I think is related to the #6484 issue - https://git.riper.fr/ext/esp8266/commit/d8531cb2c4aec39d917fed8d642afe86af59e1c6. It distinctly says they had only seen 2 nested calls, but they put in an arbitrary, hard-coded 7 to be "on the safe side". On this one exception of mine it shows 7 nested calls in three different places. Way too coincidental to not be at least part of the issue. I surmise that due to my edge-case abusive test, it actually saw seven "nestings" happening and returned an un-checked NULL pointer that finally failed as you describe.

This is what I know about my conditions:

1. This is a WebSocket library of my writing. It does not use the Arduino Core's WiFi library web socket or AsyncWebSocket. I am as close to the metal as I can get using lwIP's tcp_write() method (at time of failure)
2. The bug only occurs at high rates... 1000 sends/second. Doesn't fail at 100 sends/second.
3. It is also random (45 minutes one time, 4 hours the next). I am assuming this might be due to external network traffic (or contention)
4. Nominally, throughout the test, the Heap memory is readily available (as the history graph shows). At time of failure, yes, the heap is near fully used. I have not determined what causes the low memory spikes (shown in the Memory History). My current assumption is my message buffering routine is filling up the heap while there is some lower level TCP ACK/NAK going on because of network contention. I currently don't allow my buffering to bring the heap below 6KB. The failure does occur at one of these low spikes.
5. As a side note, the #6484 issue talks a lot about interrupts. None of the 25K lines of code in my library has interrupts explicitly used. I can't speak for Arduino Core and Espressif code that mine might be calling.

Correction on "technically"

An exception that is writing to a malloc() pointer that failed... aka a NULL pointer looks like:

Exception 0: Illegal instruction
PC: 0x402013e5
EXCVADDR: 0x00000000

My exception (in the OP) indicates some piece of code is either trying to write to or call a function in sector 0 - Which is an Espressif sector and is off-limits to Arduino Core and us.

Exception 29: StoreProhibited: A store referenced a page mapped with an attribute that does not permit stores
PC: 0x4000e1c3
EXCVADDR: 0x00000018