Sanitycheck hardware map integration caused some tests failure.

[chen-png]

Describe the bug
use the sanitycheck hardware map option to run the whole tests directory in parallel on four boards:
reel_board, frdm_k64f, sam_e70_xplained, mimxrt1050_evk,
there are some successful tests related to timer failed(those tests should be passed if only run one board), so it can't ensure the pass rate.

in the 26 failed tests:

1. test case related to timer failed
   INFO - 163/724 frdm_k64f tests/kernel/timer/timer_api/kernel.timer
   FAILED Failed (device 7.238s)
   INFO - 240/724 reel_board tests/kernel/sched/schedule_api/kernel.scheduler.multiq FAILED Failed (device 16.314s)

for a example,
if i run sanitycheck -p reel_board -T tests command on reel_board, the tests/kernel/sched/schedule_api/kernel.scheduler.multiq can be passed,

however, if i run sanitycheck --hardware-map map4.yml -T tests/, the tests/kernel/sched/schedule_api/kernel.scheduler.multiq failed on reel board with error log:
Assertion failed at ZEPHYR_BASE/tests/kernel/sched/schedule_api/src/test_slice_scheduling.c:63: thread_tslice: ((tdelta >= expected_slice_min) && (tdelta <= expected_slice_max) && (idx == thread_idx)) is false

also the same with tests/kernel/timer/timer_api/kernel.timer on frdm_k64f, this test failed with error log:
starting test - test_timer_k_define
Assertion failed at ZEPHYR_BASE/tests/kernel/timer/timer_api/src/main.c:64: duration_expire: interval >= 100 is false
FAIL - test_timer_k_define

To Reproduce
Steps to reproduce the behavior:

1. connect four boards: reel_board, frdm_k64f, sam_e70_xplained, mimxrt1050_evk
2. sanitycheck --generate-hardware-map map4.yml
3. sanitycheck --hardware-map map4.yml --device-testing -T tests/
4. See error

Screenshots or console output
INFO - 724 test configurations selected, 996 configurations discarded due to filters.
INFO - Adding tasks to the queue...

INFO - 724/724 frdm_k64f tests/lib/fdtable/libraries.os.fdtable PASSED(device 2.674s)
INFO - 663 of 724 tests passed (96.23%), 26 failed, 35 skipped with 0 warnings in 2460.19 seconds
INFO - In total 441 test cases were executed on 4 out of total 218 platforms (1.83%)

Hardware distribution summary:

Board	ID	Counter
reel_board	OSHW000032254e4500328003ab98000684d1000097969900	132
sam_e70_xplained	ATML2637010000000000	181
frdm_k64f	0240020116B95E69EB47A3D1	190
mimxrt1050_evk	000000123456	182

Environment (please complete the following information):

• OS: fedora28
• Toolchain: zephyr-sdk-0.10.3
• Commit ID: d24e5e5

[nashif]

I am having hard time reproducing or understanding the issue above, if I run sanitycheck on the same set of boards all work as expected:

nuc01:zephyr(master): sanitycheck --hardware-map ~/map.yml --device-testing -p frdm_k64f -p reel_board -p sam_e70_xplained -p mimxrt1050_evk  -T tests/kernel/
Renaming output directory to /home/nashif/zephyrproject/zephyr/sanity-out.4
INFO    - JOBS: 8
INFO    - Building initial testcase list...
INFO    - 332 test configurations selected, 40 configurations discarded due to filters.

Device testing on:

| Platform         | ID                                               | Serial device   |
|------------------|--------------------------------------------------|-----------------|
| frdm_k64f        | 0240000032044e450039700a997b00386781000097969900 |                 |
| frdm_k64f        | 0240000026334e450015400f5e0e000b4eb1000097969900 | /dev/ttyACM6    |
| frdm_k64f        | 0240000032044e4500097009997b00386781000097969900 | /dev/ttyACM7    |
| mimxrt1050_evk   | 000000123456                                     |                 |
| mimxrt1050_evk   | 000621000000                                     | /dev/ttyACM10   |
| reel_board       | OSHW000032254e4500128002ab98002784d1000097969900 | /dev/ttyACM8    |
| reel_board       | OSHW000032254e45002d8003ab98001684d1000097969900 | /dev/ttyACM14   |
| reel_board       | OSHW000041114e45002a3006bcf000320571000097969900 |                 |
| sam_e70_xplained | ATML2637010000000000                             | /dev/ttyACM5    |

INFO    - Adding tasks to the queue...
INFO    - Total complete:  332/ 332  100%  skipped:   32, failed:    0
INFO    - 300 of 300 tests passed (100.00%), 0 failed, 32 skipped with 0 warnings in 1146.24 seconds
INFO    - In total 93 test cases were executed on 4 out of total 234 platforms (1.71%)

Hardware distribution summary:

| Board            | ID                                               |   Counter |
|------------------|--------------------------------------------------|-----------|
| reel_board       | OSHW000032254e4500128002ab98002784d1000097969900 |        60 |
| sam_e70_xplained | ATML2637010000000000                             |        75 |
| reel_board       | OSHW000032254e45002d8003ab98001684d1000097969900 |        14 |
| frdm_k64f        | 0240000026334e450015400f5e0e000b4eb1000097969900 |        65 |
| frdm_k64f        | 0240000032044e4500097009997b00386781000097969900 |        11 |
| mimxrt1050_evk   | 000621000000                                     |        75 |
nuc01:zephyr(master):

you need to check the hardware and make sure:

• boards can work and flash individually, outside of sanitycheck
• You have good USB cables
• You are using a recent Linux, USB stack on older Linux distros might be an issue as well

[chen-png]

thanks. i will check my test environment.

[chen-png]

after check my environment, and add retry-failed 2 to run this again, the test result is as expected, so close this.