Optimize ESP32S2-Saola-1 remote bitbang w/ Glasgow JTAG (OCD-289) #135
Comments
|
Breakpointing and exploring w/ lldb -- openocd -d3 -f interface/remote_bitbang.cfg -f target/esp32s2.cfg.origin /cc @gerekon |
|
Now things might get a bit more interesting with Time to connect my oscilloscope to those JTAG lines and step through the Glasgow applet it seems, I'll be right back ;) |
|
I'm not seeing anything weird on those waveforms, but after connecting the probes I'm getting the following: Every 1.0s: openocd -f interface/remote_bitbang.cfg -f target/esp32s2.cfg.orig -c 'init; reset; halt' brainstorm.local: Wed Jan 13 23:59:30 2021
Open On-Chip Debugger v0.10.0-esp32-20201202-2-g7e342cad (2021-01-02-23:47)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
Warn : Transport "jtag" was already selected
Info : FreeRTOS creation
Info : Initializing remote_bitbang driver
Info : Connecting to localhost:5555
Info : remote_bitbang driver initialized
Info : This adapter doesn't support configurable speed
Error: JTAG scan chain interrogation failed: all zeroes
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: esp32s2.cpu: IR capture error; saw 0x00 not 0x01
Warn : Bypassing JTAG setup events due to errors
Info : Listening on port 3333 for gdb connections
Error: JTAG scan chain interrogation failed: all zeroes
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: esp32s2.cpu: IR capture error; saw 0x00 not 0x01
Warn : Bypassing JTAG setup events due to errors
Error: esp32s2_soc_reset: Couldn't halt target before SoC reset
Info : remote_bitbang interface quit |
|
After flashing the firmware with the following commandline: $ ~/dev/esp-idf/tools/idf.py build && /Users/romanvg/.espressif/python_env/idf4.3_py3.9_env/bin/python3 ../../esp-idf/components/esptool_py/esptool/esptool.py -p /dev/cu.SLAB_USBtoUART -b 460800 --before default_reset --after hard_reset --chip esp32s2 write_flash --flash_mode dio --flash_size detect --flash_freq 80m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/twai_network_slave.bin
Executing action: all (aliases: build)
Running ninja in directory /Users/romanvg/dev/esp32s2_can/twai_network_slave/build
Executing "ninja all"...
[1/3] Performing build step for 'bootloader'
ninja: no work to do.
Project build complete. To flash, run this command:
/Users/romanvg/.espressif/python_env/idf4.3_py3.9_env/bin/python3 ../../esp-idf/components/esptool_py/esptool/esptool.py -p (PORT) -b 460800 --before default_reset --after hard_reset --chip esp32s2 write_flash --flash_mode dio --flash_size detect --flash_freq 80m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/twai_network_slave.bin
or run 'idf.py -p (PORT) flash'
esptool.py v3.1-dev
Serial port /dev/cu.SLAB_USBtoUART
Connecting.....
Chip is ESP32-S2
Features: WiFi, ADC and temperature sensor calibration in BLK2 of efuse
Crystal is 40MHz
MAC: 7c:df:a1:00:ab:80
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x022f
Compressed 20624 bytes to 12691...
Wrote 20624 bytes (12691 compressed) at 0x00001000 in 0.3 seconds (effective 564.1 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 103...
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.0 seconds (effective 2204.9 kbit/s)...
Hash of data verified.
Compressed 154128 bytes to 85591...
Wrote 154128 bytes (85591 compressed) at 0x00010000 in 2.0 seconds (effective 607.7 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...I'm getting this output from OpenOCD right after: $ openocd -f interface/remote_bitbang.cfg -f target/esp32s2.cfg.orig -c 'init; reset; halt;'
Open On-Chip Debugger v0.10.0-esp32-20201202-2-g7e342cad (2021-01-02-23:47)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
Warn : Transport "jtag" was already selected
Info : FreeRTOS creation
Info : Initializing remote_bitbang driver
Info : Connecting to localhost:5555
Info : remote_bitbang driver initialized
Info : This adapter doesn't support configurable speed
Info : JTAG tap: esp32s2.cpu tap/device found: 0xffffffff (mfg: 0x7ff (<invalid>), part: 0xffff, ver: 0xf)
Warn : JTAG tap: esp32s2.cpu UNEXPECTED: 0xffffffff (mfg: 0x7ff (<invalid>), part: 0xffff, ver: 0xf)
Error: JTAG tap: esp32s2.cpu expected 1 of 1: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Warn : Unexpected idcode after end of chain: 160 0x1fffffff
Warn : Unexpected idcode after end of chain: 192 0x00000000
Warn : Unexpected idcode after end of chain: 224 0xf0000000
Warn : Unexpected idcode after end of chain: 320 0x0000ffff
Warn : Unexpected idcode after end of chain: 352 0x00000000
Warn : Unexpected idcode after end of chain: 384 0x00000000
Warn : Unexpected idcode after end of chain: 416 0xfe000000
Warn : Unexpected idcode after end of chain: 480 0x00ffffff
Warn : Unexpected idcode after end of chain: 512 0x00000000
Warn : Unexpected idcode after end of chain: 544 0x00000000
Warn : Unexpected idcode after end of chain: 576 0x00000000
Warn : Unexpected idcode after end of chain: 608 0xffc00000
Error: double-check your JTAG setup (interface, speed, ...)
Error: Trying to use configured scan chain anyway...
Error: esp32s2.cpu: IR capture error; saw 0x00 not 0x01
Warn : Bypassing JTAG setup events due to errors
Info : Listening on port 3333 for gdb connections
Info : JTAG tap: esp32s2.cpu tap/device found: 0xffffffff (mfg: 0x7ff (<invalid>), part: 0xffff, ver: 0xf)
Warn : JTAG tap: esp32s2.cpu UNEXPECTED: 0xffffffff (mfg: 0x7ff (<invalid>), part: 0xffff, ver: 0xf)
Error: JTAG tap: esp32s2.cpu expected 1 of 1: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: auto0.tap tap/device found: 0x7fffffff (mfg: 0x7ff (<invalid>), part: 0xffff, ver: 0x7)
Info : TAP auto1.tap has invalid IDCODE (0x0)
Info : TAP auto2.tap has invalid IDCODE (0x0)
Info : TAP auto3.tap has invalid IDCODE (0x0)
Info : TAP auto4.tap has invalid IDCODE (0x0)
Info : TAP auto5.tap has invalid IDCODE (0x0)
Info : TAP auto6.tap has invalid IDCODE (0x0)
Info : TAP auto7.tap has invalid IDCODE (0x0)
Info : TAP auto8.tap has invalid IDCODE (0x0)
Info : TAP auto9.tap has invalid IDCODE (0x0)
Info : TAP auto10.tap has invalid IDCODE (0x0)
Info : TAP auto11.tap has invalid IDCODE (0x0)
Info : TAP auto12.tap has invalid IDCODE (0x0)
Info : TAP auto13.tap has invalid IDCODE (0x0)
Info : TAP auto14.tap has invalid IDCODE (0x0)
Info : TAP auto15.tap has invalid IDCODE (0x0)
Info : TAP auto16.tap has invalid IDCODE (0x0)
Info : TAP auto17.tap has invalid IDCODE (0x0)
Info : TAP auto18.tap has invalid IDCODE (0x0)
Info : TAP auto19.tap has invalid IDCODE (0x0)
Warn : Unexpected idcode after end of chain: 83 0x00000000
Warn : Unexpected idcode after end of chain: 115 0x00000000
Warn : Unexpected idcode after end of chain: 147 0xfc000000
Warn : Unexpected idcode after end of chain: 243 0x000001ff
Warn : Unexpected idcode after end of chain: 275 0x00000000
Warn : Unexpected idcode after end of chain: 307 0xffffff00
Warn : Unexpected idcode after end of chain: 371 0x0000007f
Warn : Unexpected idcode after end of chain: 403 0x00000000
Warn : Unexpected idcode after end of chain: 435 0x00000000
Warn : Unexpected idcode after end of chain: 467 0x00000000
Warn : Unexpected idcode after end of chain: 499 0xffffffe0
Warn : Unexpected idcode after end of chain: 563 0x0000000f
Warn : Unexpected idcode after end of chain: 595 0x00000000
Warn : Unexpected idcode after end of chain: 627 0x00000000
Error: double-check your JTAG setup (interface, speed, ...)
Error: Trying to use configured scan chain anyway...
Error: esp32s2.cpu: IR capture error; saw 0x00 not 0x01
Warn : Bypassing JTAG setup events due to errors
Info : esp32s2: Debug controller was reset.
Info : esp32s2: Core was reset.
Error: esp32s2_soc_reset: Couldn't halt target before SoC reset
Info : remote_bitbang interface quit |
|
For some reason I was having some mid-level voltages and other artifacts, after reviewing the cables once more the waveforms look a bit better (Light-blue: TCK, Yellow: TDO, Dark Blue: TDI, Green: TMS):
But I'm still getting unexpected messages... the target flashes and seems to run the app well anyway, so I'm not sure if this is a symptom of a damaged JTAG pins/port?: |
|
Hm, setting up the Glasgow at 2.9V instead of 3.3V:
It almost works... OpenOCD struggles a bit though: But on the GDB side, things start to finally make sense!: |
github-actions
bot
changed the title
|
It seems that the problems are definitely with JTAG communication. We have not tested JTAG debugging with remote bitbang driver. Due to its nature there can be problems with voltage levels, timings and so on. I think this can be solved, but even solved it will work much slower then HW JTAG adapters. |
|
@brainstorm |
|
Thanks @gerekon, I understand your concerns, but I'm about to fix this for good, it's just a wiring issue:
I think that even if it's slower, it's a very accessible way to debug since the Glasgow explorer is as handy (perhaps eventually as widespread) as Raspberry Pis? I personally don't have any other HW JTAG adapter... Perhaps a Blackmagic probe would do? |
|
Besides, I do not plan to flash via JTAG (I'll do via UART instead), so speed shouldn't really be an issue for backtrace printing and some stepping and variable inspection I reckon? Or is it in your experience @gerekon ? |
|
The funky waveform shape on TDO was due to a missing 10k pullup, configurable on Glasgow this way: diff --git a/software/glasgow/applet/interface/jtag_openocd/__init__.py b/software/glasgow/applet/interface/jtag_openocd/__init__.py
index ccbf7d4..647640f 100644
--- a/software/glasgow/applet/interface/jtag_openocd/__init__.py
+++ b/software/glasgow/applet/interface/jtag_openocd/__init__.py
@@ -100,6 +100,10 @@ class JTAGOpenOCDApplet(GlasgowApplet, name="jtag-openocd"):
parser.add_argument(
"-f", "--frequency", metavar="FREQ", type=int, default=100,
help="set TCK frequency to FREQ kHz (default: %(default)s)")
+ parser.add_argument(
+ "--pulls", default=False, action="store_true",
+ help="enable integrated pull-ups")
+
def build(self, target, args):
self.mux_interface = iface = target.multiplexer.claim_interface(self, args)
@@ -111,7 +115,11 @@ class JTAGOpenOCDApplet(GlasgowApplet, name="jtag-openocd"):
))
async def run(self, device, args):
- return await device.demultiplexer.claim_interface(self, self.mux_interface, args)
+ pulls = set()
+ if args.pulls:
+ pulls = {args.pin_tdo}
+ return await device.demultiplexer.claim_interface(self, self.mux_interface, args,
+ pull_high=pulls)
@classmethod
def add_interact_arguments(cls, parser):Now it's "fixed" (still looks pretty horrid on the last screenshot, but barely within spec)... on a later on I fix it with pullups between 800ohm and 1k:
Although there might be speed problems, since I'm still seeing some spurious errors like the ones @gerekon pointed earlier, namely: Potentially tricky bit is that for bitbang, OpenOCD goes like: so even if I put Glasgow and OpenOCD at 100kHz (current default on both sides), I'm not entirely sure this is respected at all 😕 I'd like to pursue this till the end, it's more of a challenge/hobby at this point ;) |
|
Aand bingo! Going up to 500kHz (instead of the default 100kHz, I don't get any errors anymore!): And the GDB session seems to be working fine now: Finally! :D |
|
By the way, now that I got it working (relatively slow, but working), I can reproduce exactly this seemingly unresolved 2019 OpenOCD issue where @igrr seems to be involved and After launching OpenOCD I'm getting the following on GDB on a "fresh" session: The thing works "fine" after ignoring those errors though: One interesting observation is that on the OpenOCD terminal, the flash commands, presumably the ones defined in Do take a while to complete (5-15 seconds)... why does OpenOCD take so long to enumerate the memory regions?: And then the Hope that helps diagnose the problem from 2019 on the forum and get it fixed for good? |
|
After putting a "proper" (for quick-build strip-board standards) ground plane... I'll put together a proper board later, just wanted to have signals in better shape:
Also added two 820ohm pullups for both TDO & TCK the JTAG and signals look much better... admittedly not super clean, but presumably within spec?
But the esp32.com forum errors described above keep creeping up anyway, here's the openocd_log.txt Namely: As mentioned above, the memory regions enumeration takes a few seconds (is this normal?): After a subsequent Any ideas on what's the underlying issue? Happy to pursue this further myself, hints welcome ;) The esp32 forum post had exactly the same symptoms back in 2019. |
|
Predictably, some time is spent on the host OpenOCD side under
And here is the raw tracing file if you want to interactively explore in OSX's |
brainstorm
changed the title
|
Here's a bit more detail on how OpenOCD spends time... seems to be waiting on the JTAG execute queue while reading data from the IC:
|
Based on your description, it seems that the GDB server (OpenOCD) is actually not responding in time. Can you try adding |
OpenOCD needs to execute some code on the target to determine the memory mappings. This involves "running an algorithm" — in OpenOCD terms, loading a bit of code and executing it. Each time all the CPU registers need to be saved and restored. From what I remember while working with OpenOCD remote protocols, this takes significant amount of time over slow connections. The options I see are:
Options 2 and 3 likely involve modifying the remote bitbang protocol. Option 4 means that you would be limited to 2 hardware breakpoints, plus software breakpoints in IRAM. Option 5 is interesting to explore, but at the moment I don't see any low hanging fruit for such optimizations. |
|
Thanks Ivan! :D The If you look at the screenshot above for IMO, points 2&3 would yield the most bang for the buck in terms of optimization and UX improvement for users and community at large... point 5 is also interesting, but I'm not that well versed in Xtensa/Espressif intricacies so I'll just trust your judgement on that ;) ... the rest of the points are either impractical or have downsides. My objective with this issue has morphed into finding a proper fix for the root cause since I've just timed the "running an algorithm" section: ... and it takes 14 seconds to complete. This means that your own VSCode ESP-IDF extension reasonable default of 10 seconds gives up just 4 seconds short of establishing a functional debug session out of the box... it's highly possible that other IDEs behave on a similar way, generating support tickets on this repo (I've seen some similarities), so this could be a case of helping myself, you and solve a bunch of stagnant and future support requests all in one go ;) |
|
@brainstorm I understand your desire for performance improvements of remote_bitbang driver. I have checked the list of debugging-related issues reported, and only found mention of We have looked briefly into the different protocols for encapsulating JTAG transfers, and ended up implementing 2 protocols for our future devices: esp_usb_jtag, which is optimized for being rather simple to implement in hardware, and esp_remote, which can work over USB or TCP, and is more suited for a software implementation. Both protocols address shortcomings of |
|
@brainstorm Is there anything we can help to close this issue? |
|
If the remote_bitbang is in, I'm more than happy to close ;) |
As commented in the issue originated in #134 and my subsequent attempts under:
espressif/vscode-esp-idf-extension#285 (comment)
The first error in the debug logs is:
Error: 340 5543 target.c:587 target_halt(): Target not examined yetI've added the
gdb-attachbehavior since it was missing in theesp32s2.cfgopenocd target like so:But that doesn't seem to help, I've also noticed other issues that were hitting similar symptoms but none of the solutions proposed (if any) seem to help in my case (i.e #65, #69, #73, #105, etc...)
I also triple-checked the JTAG wiring (seems to be fine since Glasgow's
jtag-probeapplet correctly identifies the Tensilica JTAG ID):openocd_log_boot_mode.txt
openocd_log_app_mode.txt
gdb_log.txt
gdbinit.txt
The text was updated successfully, but these errors were encountered: