esp32: enable JTAG debugging

[zephyrbot]

Reported by Andrew Boie:

We are going to need to debug low-level boot of ESP32. This board supports OpenOCD/flyswatter. See attached document. Enable the "make debugserver" target in the build for ESP32.

(Imported from Jira ZEP-2109)

[zephyrbot]

by Andrew Boie:

• jtag_debugging_for_esp32_en.pdf (size: 1720318)

[zephyrbot]

by Leandro Pereira:

• [Flyswatter2 - 20 Pin JTAG Connector.pdf](https://s3.amazonaws.com/logs.zephyrproject.org/jira/GH-3550/Flyswatter2 - 20 Pin JTAG Connector.pdf) (size: 18151)

[zephyrbot]

by Leandro Pereira:

Here's a pinout diagram for the ESP32 board. Use this information to connect to a Flyswatter2 (JTAG pin information attached to this issue): https://hydrabus.com/2016/10/29/how-to-debug-esp32-with-jtag-openocd-gdb-1st-part-connect-the-hardware/

[zephyrbot]

by Leandro Pereira:

• IMG_20170505_142429123_HDR.jpg (size: 3532630)

[zephyrbot]

by Leandro Pereira:

• IMG_20170505_142449696_HDR.jpg (size: 3343225)

[zephyrbot]

by Leandro Pereira:

I confirm the instructions to attach a JTAG are working. OpenOCD is running and it can communicate with the ESP32 board. Not sure yet how useful this will be to debug early boot issues yet, though.

[zephyrbot]

by Leandro Pereira:

OpenOCD recognizes the SoC (not always), but cannot do anything (inspect/modify memory, breakpoints, control execution, get register dump, etc). Still don't know why. Tested with different ESP32 boards, and different Flyswatter2 devices.

[zephyrbot]

by Andrew Boie:

That's interesting. Are you able to debug FreeRTOS using their tools? This might be a good topic for the call with Espressif.

[zephyrbot]

by Leandro Pereira:

• IMG_20170605_090928338.jpg (size: 3683825)

[zephyrbot]

by Leandro Pereira:

• IMG_20170605_090905672_HDR.jpg (size: 4022041)

[zephyrbot]

by Leandro Pereira:

I got JTAG to work reliably now, even without FreeRTOS. It's slightly slow, but better than shooting in the dark. This was accomplished by building a small adapter board that the ESP32 board can be connected to:

!IMG_20170605_090928338.jpg|thumbnail!

It's a convenient board, as it snaps directly into the Flyswatter connector, connecting all the required signals at the same time, and using short wires (and thus reducing possibility for interference). There's very little room for error when connecting or disconnecting the board to the JTAG.

!IMG_20170605_090905672_HDR.jpg|thumbnail!

The above picture isn't up-to-date, as the Vref and Gnd signals (which the documentation didn't mention was required) are now also connected. In addition, two rows of pin headers are also soldered to each side, making it easier to plug things to the board while it's connected to the JTAG; useful to debug I/O.

[zephyrbot]

by Leandro Pereira:

I'm leaving this issue open for the moment, since most of these comments and pictures will be published as documentation (no code has been produced to implement this story).

[zephyrbot]

by Leandro Pereira:

#788

[zephyrbot]

by Mark Linkmeyer:

Inaky Perez-Gonzalez , as part of setting up ESP32 for TCF, can you verify this one? Thx.

[zephyrbot]

by Inaky Perez-Gonzalez:

Once the boards are done, I will experiment with converting one of the boards in the setup to be openocd based

[zephyrbot]

by Mark Linkmeyer:

Hi Andy Ross , in any of the work you're doing to help provide backup for Leandro who is on vacation, have you observed that JTAG debugging is working? If so, is that sufficient for the purposes of verify this story? I'd think so. If so, can you move this to Closed (as verified)? Thanks.

[zephyrbot]

by Mark Linkmeyer:

Andy Ross to verify.

[zephyrbot]

by Mark Linkmeyer:

Andy Ross , please reply to questions/comments above!! It's been 2-3 weeks.

[zephyrbot]

by Mark Linkmeyer:

Per email Mark received from Andy, "This is waiting on hardware. Just sent another ping to the lab guys. It's supposed to be done, just need to find the PCBs."

[zephyrbot]

by Andy Ross:

Leandro loaned me his (very slick) breakout prototype. This almost works:

I can get the Espressif openocd built and connected just fine. Commands like "reset" and "halt" work as expected, but the gdbserver (nothing that has anything to do with the hardware work tracked here) seems to be speaking the wrong bits:

cd ~/esp/openocd-esp32
sudo src/openocd -s tcl -f interface/ftdi/flyswatter2.cfg -f board/esp-wroom-32.cfg 

/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/xtensa-zephyr-elf/xtensa-zephyr-elf-gdb
(gdb) target remote localhost 3333
Remote debugging using localhost:3333
Remote 'g' packet reply is too long: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
(gdb) ba
No stack.

It's worth pointing out that the system does halt when gdb connects, so something is working.

What am I doing wrong? This really feels like it should be user error, openocd and gdb have been working together since the beginning of time.

[zephyrbot]

by Andy Ross:

Ah, OK. Leandro points out that you must use the gdb distributed by Espressif, our SDK gdb won't work (have they seriously hacked the gdbserver protocol?).

And now it connects, and we're almost there. I can read out memory state just fine. Unfortunately it's not able to stop at breakpoints in the zephyr binary. Looking carefully, gdb is reporting a $PC value of 0xf8001080, which matches the (halted) PC of the APP_CPU reported by openocd, and not the PRO_CPU on which Zephyr runs (the firmware halts the app cpu at boot and we never enable it).

I can't find a reference in the Espressif docs to managing the two CPUs. Is this just not possible right now or am I missing something?

If not, I'd be tempted to close this as verified given that this is doing 80% of what we want a debugger to do now.

[zephyrbot]

by Andy Ross:

OK! Finally have this working. The magic was the ESP_ONLYCPU setting, which Leandro had found in a comment in the board config file. So with a current openocd-esp32 from Espressif's github and the current SDK installed in ~/esp:

~/esp/openocd-esp32$ sudo src/openocd -f interface/ftdi/flyswatter2.cfg -c 'set ESP32_ONLYCPU 1' -c 'set ESP32_RTOS none' -f board/esp-wroom-32.cfg -s tcl

# And in another shell:
~/zephyr/samples/hello_world$ ~/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-gdb outdir/esp32/zephyr.elf

(gdb) target remote localhost:3333

Remote debugging using localhost:3333
0x40082056 in k_cpu_idle ()
    at /home/ajross/z/zephyr/arch/xtensa/core/cpu_idle.c:22
22		__asm__ volatile ("waiti 0");

(gdb) ba

<span>#</span>0  0x40082056 in k_cpu_idle ()
    at /home/ajross/z/zephyr/arch/xtensa/core/cpu_idle.c:22
<span>#</span>1  0x40082101 in _sys_power_save_idle (ticks=<optimized out>)
    at /home/ajross/z/zephyr/kernel/idle.c:122
<span>#</span>2  idle (unused1=0x0, unused2=0x0, unused3=0x0)
    at /home/ajross/z/zephyr/kernel/idle.c:169
<span>#</span>3  0x400818f9 in _thread_entry (entry=0x400820f8 <idle>, p1=0x0, p2=0x0, 
    p3=0x0) at /home/ajross/z/zephyr/kernel/thread.c:186
Backtrace stopped: Cannot access memory at address 0xc0c73e85

We should probably fix up the docs some, as Espressif don't actually call this variable out anywhere in their own whitepaper. But it works.