Better crash dump reporting
Posted: Sat Apr 04, 2020 6:50 pmI have been using the ESP RTOS SDK but I recently switched to using the Arduino SDK. One of the best things about the RTOS SDK is its crash logging. It has something called IDF Monitor which is a tool that acts as a serial console, but has some special behavior when it sees that the device has crashed. The raw crash log looks like this:
but IDF Monitor decorates it to make it look like this:
All those code pointers use ASCII colors so they stand out better.
You give IDFMonitor a pointer to the elf file and it uses addr2line to find any hex values that are in code space and prints the line in a source file. It makes it super easy to see where the crash occurred, or at least near where it crashed.
You can use IDF Monitor as the console for Arduino, but it doesn't give nearly as useful info. It might give you the line of code for the current PC, but that's usually an abort or assert of something. The problem is that the crash dump generated by Arduino doesn't format it's hex numbers with a 0x at the start and it doesn't seem to dump the entire stack trace.
Is there a similar mechanism to make crash debugging easier on Arduino or is there a replacement crash dump output that is more useful?
Code: Select all
Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC : 0x402234de PS : 0x00000033 A0 : 0x402234db A1 : 0x3ffef640
A2 : 0x00000001 A3 : 0x3ffea65c A4 : 0xffffffff A5 : 0x00000000
A6 : 0x3fff4838 A7 : 0x3fff4830 A8 : 0xbfff4880 A9 : 0x0000012c
A10 : 0x00000006 A11 : 0x00000002 A12 : 0x00000000 A13 : 0x00000000
A14 : 0x00000000 A15 : 0x3ffef680 SAR : 0x0000001f EXCCAUSE: 0x0000001c
Backtrace: 0x402234de:0x3ffef640 0x40225e88:0x3ffef680 0x40226389:0x3ffef6a0 0x4025e978:0x3ffef6c0 0x4025ef89:0x3ffef6e0 0x40260454:0x3ffef700 0x40260514:0x3ffef760 0x402600bb:0x3ffef770
but IDF Monitor decorates it to make it look like this:
Code: Select all
Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC : 0x402234de PS : 0x00000033 A0 : 0x402234db A1 : 0x3ffef640
0x402234de: xQueueGenericSend at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/freertos/freertos/queue.c:2332
0x402234db: xQueueGenericSend at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/freertos/freertos/queue.c:2332
A2 : 0x00000001 A3 : 0x3ffea65c A4 : 0xffffffff A5 : 0x00000000
A6 : 0x3fff4838 A7 : 0x3fff4830 A8 : 0xbfff4880 A9 : 0x0000012c
A10 : 0x00000006 A11 : 0x00000002 A12 : 0x00000000 A13 : 0x00000000
A14 : 0x00000000 A15 : 0x3ffef680 SAR : 0x0000001f EXCCAUSE: 0x0000001c
Backtrace: 0x402234de:0x3ffef640 0x40225e88:0x3ffef680 0x40226389:0x3ffef6a0 0x4025e978:0x3ffef6c0 0x4025ef89:0x3ffef6e0 0x40260454:0x3ffef700 0x40260514:0x3ffef760 0x402600bb:0x3ffef770
0x402234de: xQueueGenericSend at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/freertos/freertos/queue.c:2332
0x40225e88: sys_mbox_post at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/lwip/port/esp8266/freertos/sys_arch.c:408
0x40226389: tcpip_api_call at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/lwip/lwip/src/api/tcpip.c:408
0x4025e978: tcpip_adapter_start_dhcp at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/tcpip_adapter/tcpip_adapter_lwip.c:1220
0x4025ef89: tcpip_adapter_dhcpc_start at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/tcpip_adapter/tcpip_adapter_lwip.c:1220
0x40260454: system_event_sta_connected_handle_default at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/esp8266/source/event_default_handlers.c:126
0x40260514: esp_event_process_default at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/esp8266/source/event_default_handlers.c:308
0x402600bb: esp_event_loop_task at /Users/cmarrin/esp/ESP8266_RTOS_SDK/components/esp8266/source/event_loop.c:92
All those code pointers use ASCII colors so they stand out better.
You give IDFMonitor a pointer to the elf file and it uses addr2line to find any hex values that are in code space and prints the line in a source file. It makes it super easy to see where the crash occurred, or at least near where it crashed.
You can use IDF Monitor as the console for Arduino, but it doesn't give nearly as useful info. It might give you the line of code for the current PC, but that's usually an abort or assert of something. The problem is that the crash dump generated by Arduino doesn't format it's hex numbers with a 0x at the start and it doesn't seem to dump the entire stack trace.
Is there a similar mechanism to make crash debugging easier on Arduino or is there a replacement crash dump output that is more useful?