Add a CLI version of ESPExceptionDecoder

[earlephilhower]

Seems like Platform.IO doesn't have JAVA plugins, so running the stack dump decoder is not an option for them.

It would be nice to have a CLI version of the exception decoder that would take the ELF file and stack dump as input and produce the same output as the java plugin.

[liebman]

This CLI decoder has worked for me in the past https://github.com/littleyoda/EspStackTraceDecoder

[earlephilhower]

Thanks @liebman. It needs a little updating (it still uses addr2line which has issues with reporting on the Xtensa architecture and is missing the last failed allocation parsing) but OTW looks like something I can add to the toolchain build and be part of the installed binaries easily enough!

[liebman]

I was just looking at it as well and came across this one (I've not used it): https://github.com/luffykesh/EspExceptionDecoder-CLI

[liebman]

I was also thinking it should not be too hard to implement in python which may be easier for PlatformIO and other non-java based tool sets.

[earlephilhower]

Jackpot on #6310 (comment). It's a fork of the GUI with the GUI bits stripped off. It has my updates to use GDB and parse the OOM error already, so that's a plug-and-play one.

A Python or even a compiled C++ app would also be pretty trivial, like you suggested. Really, it is just a wrapper for gdb, so parsing the text and finding the PC/EXCVPC/STACK/OOM-Line are the only complicated tasks.

[igrr]

addr2line which has issues with reporting on the Xtensa architecture

FWIW if you update the binutils to a recent version (2.31), the issue will be solved. That version also improves objdump output by taking instruction padding into account (https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=4b8e28c79356265b2c111e044142fb6d6d2db44e).

[earlephilhower]

Good to see you still around, @igrr. Looks like we're OK to use addr2line:

2.5.2 release:

earle@server:~/Arduino/hardware/esp8266com/esp8266/tools$ ./xtensa-lx106-elf/bin/xtensa-lx106-elf-addr2line -v
GNU addr2line (GNU Binutils) 2.31.51.20180723
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

And we're at whatever GNU head is for the GCC 9.1 compiler chain:

earle@server:~/Arduino/hardware/esp8266com/esp8266/tools$ ./xtensa-lx106-elf/bin/xtensa-lx106-elf-addr2line -v
GNU addr2line (GNU Binutils) 2.32.51.20190713
Copyright (C) 2019 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

[mcspr]

Just as an additional info, this small python script does the decoding using addr2line:
https://github.com/janLo/EspArduinoExceptionDecoder
(and https://github.com/me21/EspArduinoExceptionDecoder is the fork with Windows compatibility fixes)
No OOM parser though. But I can confirm that it does work fine with stack trace part, unlike the old toolchain with binutils-2.24.

Also, ref ESP8266 platform issue: platformio/platform-espressif8266#31

How would it be expected to be used though? PlatformIO cli supports additional custom "targets" (like -t checkprogsize, -t nobuild -t upload etc.), which can be configured per build environment. IDK how that would look in VSCode.
Or, one interesting option is to do something like idf_monitor.py (ref platformio/platform-espressif32#105 (comment)) and decode stack trace right when it happens in the serial monitor app (pio device monitor)

[earlephilhower]

Wow, a CLI decoder seems popular! It's trivial to add a pre-existing CLI tool to the release, but I'm definitely not the one to choose it since it seems other folks actually are running into problems.

Just on the fact that it's Python (and we have Python on Win in our tools dir) and cross platform, I'm inclined to either include https://github.com/me21/EspArduinoExceptionDecoder or list it in the readme so others can find it.

But PIO integration or VS.code, that seems like it needs to be handled in the respective repos, no?

[Niek]

Here's another vote for including the CLI decoder.

[d-a-v]

So which one(s) should we include as a submodule and document ?

1. https://github.com/janLo/EspArduinoExceptionDecoder
2. https://github.com/me21/EspArduinoExceptionDecoder (PR: Fixed running on Windows. Added ESP32 backtrace parsing. janLo/EspArduinoExceptionDecoder#5)
3. https://github.com/littleyoda/EspStackTraceDecoder
4. https://github.com/luffykesh/EspExceptionDecoder-CLI
5. Backtrace decoder for ESP32 platformio/platform-espressif32#105 (comment) (aka idf_monitor)

[everslick]

idf_monitor seems to be the most user friendly to me. All others need some kind of manual work to load/pipe the stacktrace into the decoder.

If it ain't idf_monitor for any reason, then I'd prefer a python script to a java application. we have to have python anyway already.

[TD-er]

I tested nr 2 today (see #6574 ) and just a few remarks:

• I didn't see line number (tested in Windows) in the output.
• Had to change the order of lines in the stack dump output before the decoder could parse it.

About the line order in the stack dump; Not sure if it was an unintended change in how the stack dump is printed, but I had to move the >>>stack>>> line to right above the actual dump.
When taken from the serial dump, it is printed above the lines

ctx: cont
sp: ......

[earlephilhower]

I also didn't get any line numbers on Linux with number 2. I didn't look into it, but it should be a simple fix.

The problem with idf-monitor (number 5) is that it seems to me that you can't cut-n-paste stack dumps into it since it's trying to connect to a serial port and be the master interface (vs. screen or cu or Putty/PowerTerm on Windows). Can idf-monitor.py take a snippet somehow from a file or cut-n-paste? We already have Python and PySerial installed on Windows in a private portable dir, so that's not a problem.

[devyte]

Pushing this back.

[d-a-v]

I tried idf-monitor with this "segfault.ino" sketch:

void setup() {
  Serial.begin(115200);
  delay(10000);
  *((int*)0) = 0;
}

void loop() {
}

and this command-line:

python3 idf_monitor.py --port /dev/ttyUSB0  --baud 115200 /tmp/arduino_build_460962/segfault.ino.elf --toolchain-prefix $ESP8266ARDUINO/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-

and the result was:

SDK:2.2.2-dev(5ab15d1)/Core:2.6.1-1-g02eabaae=20601001/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-16-ge23a07e/BearSSL:89454af
Fatal exception 29(StoreProhibitedCause):
epc1=0x40201041, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000
0x40201041: setup at /home/gauchard/dev/proj/arduino/segfault/segfault.ino:5

Exception (29):
epc1=0x40201041 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000
0x40201041: setup at /home/gauchard/dev/proj/arduino/segfault/segfault.ino:5

>>>stack>>>
ctx: cont
sp: 3ffffe00 end: 3fffffc0 offset: 01a0
3fffffa0:  feefeffe feefeffe feefeffe 40201bcc  
3fffffb0:  feefeffe feefeffe 3ffe84d8 401012e9  
<<<stack<<<

On bare serial console it is:

SDK:2.2.2-dev(5ab15d1)/Core:2.6.1-1-g02eabaae=20601001/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-16-ge23a07e/BearSSL:89454af
Fatal exception 29(StoreProhibitedCause):
epc1=0x40201041, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

Exception (29):
epc1=0x40201041 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3ffffe00 end: 3fffffc0 offset: 01a0
3fffffa0:  feefeffe feefeffe feefeffe 40201bcc  
3fffffb0:  feefeffe feefeffe 3ffe84d8 401012e9  
<<<stack<<<

The stack does not seem to be decoded.

[mcspr]

I don't think it should? IDF has a different output for backtrace

[devyte]

Reducing 2.7.0 scope, pushing back

[d-a-v]

Someone with a working solution or feedback post any findings !

[TD-er]

FYI.
In PlatformIO with the recent add-on of the exception decoder filter, I how have a working stack decoding.

In PlatformIO.ini:

platform = [email protected]
monitor_filters = esp8266_exception_decoder

It still only works while dev'ing, so I cannot use it for user provided stack dumps, but at least it is a (very good) start.

[d-a-v]

Another interesting path:
Using https://raw.githubusercontent.com/espressif/esp-idf/master/tools/idf_monitor.py as serial console replacement live-translate on-console addresses in the form of 0x4xxxxxxx to <function> at <file>:<line>:

SDK:2.2.2-dev(38a443e)/Core:2.7.2-60-gc1117132=20702060/lwIP:IPv6+STABLE-2_1_2_RELEASE/glue:1.2-31-g3798bf0/BearSSL:149e503
wait 5s
51328
Fatal exception 29(StoreProhibitedCause):
epc1=0x4020109b, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000
0x4020109b: loop at /path/to/new.ino:25       <------- this line is added by idf_monitor

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (29):
epc1=0x4020109b epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000
0x4020109b: loop at /path/to/new.ino:25       <------- this line is added by idf_monitor


>>>stack>>>

ctx: cont
sp: 3ffffe00 end: 3fffffc0 offset: 0190
3fffff90:  3fffdad0 00000000 3ffee5f8 40201091  
3fffffa0:  feefeffe feefeffe 3ffee5f8 402023fc  
3fffffb0:  feefeffe feefeffe 3ffe84f8 40101295  
<<<stack<<<
--------------- CUT HERE FOR EXCEPTION DECODER ---------------

(the line is indeed: /path/to/new.ino:25: *(int*)0 = 0;)

By using

curl https://raw.githubusercontent.com/espressif/esp-idf/master/tools/idf_monitor.py > idf_monitor.py
python3 /path/to/idf_monitor.py
        --port /dev/ttyUSB0
        --baud 115200
        --toolchain-prefix /path/to/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-
        /tmp/new-d1/new.ino.elf

instead of

miniterm.py /dev/ttyUSB0 115200

That may be interesting with makeEspArduino @plerup

[d-a-v]

Before pushing to a further release, or closing,
Does someone have any news or use something else on that matter ?

[d-a-v]

You are welcome to help with #8763

[hmueller01]

This is closed, but I think it's a good place to ask here.

Is it necessary to build a debug environment (build_type = debug) to get all details from the decoder?