edbg: long lines flooded over serial become garbled #14548

bergzand · 2020-07-17T16:26:28Z

Description

The virtual COM port provided by the EDBG debuggers doesn't correctly handled rapid bulk transfers from the host computer. This becomes apparent when using a local setup. A remote setup or a slow host such as a raspberry pi doesn't flood the USB serial fast enough to cause issues. That's also why it doesn't fail on the nightlies.

Bisecting turns up this commit: 68a35f5, but that only uncovered the issue.

Steps to reproduce the issue

Locally run tests/shell on a board using EDBG such as the samr21-xpro or the same54-xpro. This fails on the long line test, where it starts to produce garbage chars.

Expected results

The test succeeds both locally and on Murdock/IoT-LAB.

Actual results

The test fails after the EDBG serial produces garbage instead of the long line:

local node FAIL

$ BOARD=samr21-xpro make -C tests/shell/ flash test
make: Entering directory '/home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell'
Building application "tests_shell" for "samr21-xpro" with MCU "samd21".

"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/boards/samr21-xpro
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/core
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/samd21
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/cortexm_common
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/cortexm_common/periph
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/sam0_common
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/sam0_common/periph
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/samd21/periph
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/drivers
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/drivers/periph_common
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/app_metadata
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/auto_init
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/isrpipe
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/newlib_syscalls_default
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/pm_layered
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/ps
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/shell
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/shell/commands
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/stdio_uart
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/tsrb
   text	   data	    bss	    dec	    hex	filename
  12848	    132	   2624	  15604	   3cf4	/home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/samr21-xpro/tests_shell.elf
/home/mlenders/Repositories/RIOT-OS/RIOT-release/dist/tools/edbg/edbg  --target samr21 --verbose --file /home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/samr21-xpro/tests_shell.bin --verify || /home/mlenders/Repositories/RIOT-OS/RIOT-release/dist/tools/edbg/edbg  --target samr21 --verbose --file /home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/samr21-xpro/tests_shell.bin --verify --program
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800005161 01.1A.00FB (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Verification...................................................... done.

socat - open:/dev/ttyACM0,b115200,echo=0,raw
�	"R�
shell: incorrect quoting
> bufsize
x�
> 
> bufsize
128
> ________________________________________________________________________________________________________________________________verylong
________________________________________________________________�8����A 8
                                                                         	$�`�@��^�吧���!:۳
shell: command not found: ________________________________________________________________�8����A
> -j1/��$n�H�"<P)�I�{mː�v�?DnTimeout in expect script at "child.expect('shell: maximum line length exceeded')" (../RIOT/tests/shell/tests/01-run.py:150)

make: *** [/home/mlenders/Repositories/RIOT-OS/RIOT-release/Makefile.include:770: test] Error 1
make: Leaving directory '/home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell'

However, it doesn't fail when using the same setup on the IoT-LAB:

On Saclay site of IoT-LAB PASS

$ BOARD=samr21-xpro IOTLAB_NODE=auto-ssh make -C tests/shell/ flash test
make: Entering directory '/home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell'
Building application "tests_shell" for "samr21-xpro" with MCU "samd21".

"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/boards/samr21-xpro
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/core
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/samd21
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/cortexm_common
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/cortexm_common/periph
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/sam0_common
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/sam0_common/periph
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/cpu/samd21/periph
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/drivers
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/drivers/periph_common
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/app_metadata
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/auto_init
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/isrpipe
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/newlib_syscalls_default
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/pm_layered
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/ps
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/shell
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/shell/commands
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/stdio_uart
"make" -C /home/mlenders/Repositories/RIOT-OS/RIOT-release/sys/tsrb
   text	   data	    bss	    dec	    hex	filename
  12848	    132	   2624	  15604	   3cf4	/home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/samr21-xpro/tests_shell.elf
iotlab-node --jmespath='keys(@)[0]' --format='int'  --list saclay,samr21,10 --flash /home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/samr21-xpro/tests_shell.bin | grep 0
0

ssh -t lenders@saclay.iot-lab.info 'socat - tcp:samr21-10.saclay.iot-lab.info:20000'

> 
> bufsize
bufsize
128
> ________________________________________________________________________________________________________________________________verylong
________________________________________________________________________________________________________________________________verylong
shell: maximum line length exceeded
> garbage1234
garbage1234
> 
> start_test
start_test
[TEST_START]
> 

> 
> 


> 
> echo a string
echo a string
"echo""a""string"
> echo   multiple   spaces   between   argv
echo   multiple   spaces   between   argv
"echo""multiple""spaces""between""argv"
> echo 	 tabs		 processed 		like	 		spaces
echo 	 tabs		 processed 		like	 		spaces
"echo""tabs""processed""like""spaces"
> 123456789012345678901234567890123456789012345678901234567890
123456789012345678901234567890123456789012345678901234567890
shell: command not found: 123456789012345678901234567890123456789012345678901234567890
> unknown_command
unknown_command
shell: command not found: unknown_command
>      echo leading spaces
     echo leading spaces
"echo""leading""spaces"
> 					echo leading tabs
					echo leading tabs
"echo""leading""tabs"
> echo trailing spaces     
echo trailing spaces     
"echo""trailing""spaces"
> echo trailing tabs					
echo trailing tabs					
"echo""trailing""tabs"
> hello-world
hello-world
shell: command not found: hello-world
echo
echo
"echo"
> echo \'
echo \'
"echo""'"
> echo \"
echo \"
"echo""""
> echo escaped\ space
echo escaped\ space
"echo""escaped space"
> echo escape within '\s\i\n\g\l\e\q\u\o\t\e'
echo escape within '\s\i\n\g\l\e\q\u\o\t\e'
"echo""escape""within""singlequote"
> echo escape within "\d\o\u\b\l\e\q\u\o\t\e"
echo escape within "\d\o\u\b\l\e\q\u\o\t\e"
"echo""escape""within""doublequote"
> echo "t\e st" "\"" '\'' a\ b
echo "t\e st" "\"" '\'' a\ b
"echo""te st"""""'""a b"
> echo "hello"world
echo "hello"world
"echo""helloworld"
> echo hel"lowo"rld
echo hel"lowo"rld
"echo""helloworld"
> echo hello"world"
echo hello"world"
"echo""helloworld"
> echo quoted space " "
echo quoted space " "
"echo""quoted""space"" "
> echo abc"def'ghijk"lmn
echo abc"def'ghijk"lmn
"echo""abcdef'ghijklmn"
> echo abc'def"ghijk'lmn
echo abc'def"ghijk'lmn
"echo""abcdef"ghijklmn"
> echo "'" '"'
echo "'" '"'
"echo""'""""
> echo a\
echo a\
shell: incorrect quoting
> echo "
echo "
shell: incorrect quoting
> echo '
echo '
shell: incorrect quoting
> echo abcdef"ghijklmn
echo abcdef"ghijklmn
shell: incorrect quoting
> echo abcdef'ghijklmn
echo abcdef'ghijklmn
shell: incorrect quoting
> ps
ps
	pid | state    Q | pri 
	  1 | pending  Q |  15
	  2 | running  Q |   7
> help
help
Command              Description
---------------------------------------
bufsize              Get the shell's buffer size
start_test           starts a test
end_test             ends a test
echo                 prints the input command
reboot               Reboot the node
version              Prints current RIOT_VERSION
pm                   interact with layered PM subsystem
ps                   Prints information about running threads.
app_metadata         Returns application metadata
> reboot
reboot
main(): This is RIOT! (Version: 2020.10-devel-18-gab072-2020.07-branch)
test_shell.
> end_test
end_test
[TEST_END]
> 
make: Leaving directory '/home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell'

The text was updated successfully, but these errors were encountered:

miri64 · 2020-07-17T16:27:55Z

See also discussion in RIOT-OS/Release-Specs#172 (comment) and the following comments.

miri64 · 2020-07-17T16:44:42Z

It was also noted that this might be related to #10639

maribu · 2023-05-19T12:47:32Z

From the edbg release history:

Firmware version (decimal) Date Relevant changes

[...] [...] [...]

2.17 11.11.2015 CDC buffer fix

Looks like updating the edbg firmware to 2.17 could fix the issues. I'll spin up a Windows machine to see if updating the firmware helps. (I'm really sure that my samr21-xpro was produced after 2015, though. But you never know.)

If indeed an EDBG update fixes this issue, my suggestion would be to document this and ask uses to only use fixed EDBG firmware. If this doesn't fix the issue I see the following options:

Document this as a hardware bug and live with it
Work around the issue
- in the board --> host direction, introduce a sleep after every 48 bytes send on the stdio
- in the host --> board direction (yes, chars are lost in this direction as well) update pyterm and our test python thingy to introduce a sleep after every 48 bytes send
Prefer other stdio transports
- stdio_rtt works now fine with both JLinkExe and OpenOCD. Maybe this can be made to work with EDBG as well?
Close this as won't fix, blame Atmel/Microchip

maribu · 2023-05-19T18:23:52Z

Nope, on my samr21-xpro a freshly updated edbg firmware still won't work with long shell command lines.

bergzand added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Area: tools Area: Supplementary tools labels Jul 17, 2020

miri64 added this to the Release 2020.07 milestone Jul 17, 2020

miri64 mentioned this issue Jul 17, 2020

Release 2020.07 - RC2 RIOT-OS/Release-Specs#172

Closed

81 tasks

miri64 modified the milestones: Release 2020.07, Release 2020.10 Jul 24, 2020

MrKevinWeiss modified the milestones: Release 2020.10, Release 2021.07 Jun 22, 2021

MrKevinWeiss removed this from the Release 2021.07 milestone Jul 15, 2021

maribu mentioned this issue May 24, 2023

sys/stdio_uart: dropped data when received at once #10639

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

edbg: long lines flooded over serial become garbled #14548

edbg: long lines flooded over serial become garbled #14548

bergzand commented Jul 17, 2020

miri64 commented Jul 17, 2020

miri64 commented Jul 17, 2020

maribu commented May 19, 2023

maribu commented May 19, 2023

edbg: long lines flooded over serial become garbled #14548

edbg: long lines flooded over serial become garbled #14548

Comments

bergzand commented Jul 17, 2020

Description

Steps to reproduce the issue

Expected results

Actual results

miri64 commented Jul 17, 2020

miri64 commented Jul 17, 2020

maribu commented May 19, 2023

maribu commented May 19, 2023