Release 2022.04 - RC3

[OlegHahm]

This issue lists the status of all tests for the Release Candidate 3 of the 2022.04 release.

Specs tested:

• 01-ci
  • Task #01 - Compile test
  • Task #02 - Unittests on native @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - Unittests on native separated @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - Unittests on iotlab-m3 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
• 02-tests
  • Task #01 - All tests on native
  • Task #02 - Subset of tests on iotlab-m3
  • Task #03 - Subset of tests on samr21-xpro
• 03-single-hop-ipv6-icmp
  • Task #01 - ICMPv6 multicast echo on native @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - ICMPv6 link-local echo on native @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - ICMPv6 link-local echo on native (1 hour) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - ICMPv6 stress test on native (1 hour) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #05 - ICMPv6 stress test on native (neighbor cache stress)
  • Task #06 - ICMPv6 link-local echo on native (IPv6 fragmentation) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
• 04-single-hop-6lowpan-icmp
  • Task #01 - ICMPv6 link-local echo with iotlab-m3 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - ICMPv6 multicast echo with iotlab-m3/samr21-xpro @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - ICMPv6 echo with large payload @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - ICMPv6 echo with iotlab-m3/samr21-xpro 15 minutes @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #05 (Experimental) - ICMPv6 multicast echo with samr21-xpro/cc2538
  • Task #06 (Experimental)- ICMPv6 link-local echo with samr21-xpro/cc2538
  • Task #07 (Experimental) - ICMPv6 multicast echo with samr21-xpro/zero + xbee
  • Task #08 (Experimental) - ICMPv6 echo with samr21-xpro/zero + xbee
  • Task #09 - ICMPv6 stress test on iotlab-m3 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #10 (Experimental) - ICMPv6 echo with large payload (IPv6 fragmentation) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #11 (Experimental) - ICMPv6 stress test on nrf802154
  • Task #12 (Experimental) - ICMPv6 multicast echo with iotlab-m3/nrf802154
  • Task #13 (Experimental) - ICMPv6 link-local echo with iotlab-m3/nrf802154
• 05-single-hop-route
  • Task #01 - ICMPv6 echo unicast addresess on native (default route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - ICMPv6 echo unicast addresess on iotlab-m3 (default route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - ICMPv6 echo unicast addresess on native (specific route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - ICMPv6 echo unicast addresess on iotlab-m3 (static route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
• 06-single-hop-udp
  • Task #01 - UDP on iotlab-m3 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - UDP on iotlab-m3 (UDP port compression) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - UDP on native (non-existent neighbor) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - UDP on iotlab-m3 (non-existent neighbor) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #05 - Empty UDP on native @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #06 - Empty UDP on iotlab-m3 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
• 07-multi-hop
  • Task #01 - ICMPv6 echo on iotlab-m3 with three hops (static route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - UDP on iotlab-m3 with three hops (static route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - ICMPv6 echo on iotlab-m3 with three hops (RPL route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - UDP on iotlab-m3 with three hops (RPL route) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #05 (Experimental) - UDP with large payload on iotlab-m3 with three hops (RPL route)
• 08-interop
  • Task #01 - ICMPv6 echo between native and Linux @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - ICMPv6 echo between iotlab-m3 and Linux with 6LowPAN
  • Task #03 - UDP exchange between iotlab-m3 and Contiki
  • Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN
  • Task #05 - ICMPv6 echo between iotlab-m3 and Internet host through RIOT border router
  • Task #06 - UDP between iotlab-m3 and Internet host through RIOT border router
  • Task #07 - UDP between iotlab-m3 and Internet host through RIOT border router (200b payload)
  • Task #08 - UDP between GNRC and lwIP on iotlab-m3 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task Release 2015.12 - pre-RC1 #9 - deprecated
  • Task Release 2015.12 - RC1 #10 - deprecated
  • Task #11 - UDP exchange between iotlab-m3 and Zephyr
  • Task #12 - ICMPv6 echo between Border Router with WiFi uplink to named Internet host
  • Task #13 - ICMPv6 echo between ESP and named Internet host through Linux with 6LowPAN
• 09-coap
  • Task #01 - CORD Endpoint @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - Confirmable retries @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - Block1 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - Block2 @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #05 - Observe registration and notification @riot-ci (see Release 2022.04 - RC3 #244 (comment))
• 10-icmpv6-error
  • Task #01 - Destination unreachable - no route to destination @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #02 - Destination unreachable - Beyond scope of source address @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - Destination unreachable - address unreachable (target node address) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - Destination unreachable - address unreachable (neighbor cache miss) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #05 - Destination unreachable - port unreachable @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #06 - Destination unreachable - port unreachable (large payload) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #07 - Packet too big @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #08 - Time exceeded - hop limit exceeded in transit @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #09 - Parameter problem - erroneous header field encountered @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #10 - IPv6-in-IPv6 encapsulation @riot-ci (see Release 2022.04 - RC3 #244 (comment))
• 11-lorawan
  • Task #01 - LoRaWAN example
  • Task #02 - OTAA join procedure @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #03 - ABP join procedure @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #04 - LoRaWAN device parameters persistence @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #05 - GNRC LoRaWAN (Over The Air Activation) @riot-ci (see Release 2022.04 - RC3 #244 (comment))
  • Task #06 - GNRC LoRaWAN (ABP)
• 99-compile-and-test-one-board
  • Task #01 - Run tests on different hardwares

[riot-ci]

Test Report

	Task	Outcome
✔	01. Task 02 - Unittests on native	PASSED
✔	01. Task 03 - Unittests on native separated [tests-analog_util]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-base64]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bcd]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bitfield]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bloom]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bluetil]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-checksum]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-chunked_ringbuffer]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-clif]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-color]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-core]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-credman]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-div]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ecc]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-fib]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-fib_sr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-fmt]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gcoap]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_ipv6]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_ipv6_hdr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_ipv6_nib]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_mac_internal]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_netif_pktq]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_sixlowpan_frag_vrb]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_udp]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-hashes]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ieee802154]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-inet_csum]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ipv4_addr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ipv6_addr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ipv6_hdr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-kernel_defines]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-luid]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-matstat]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-mtd]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-nanocoap]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-netopt]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-netreg]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-phydat]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-pkt]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-pktbuf]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-pktqueue]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-printf_float]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-priority_pktqueue]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-rtc]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-rtt_rtc]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-saul_reg]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-scanf_float]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-seq]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sht1x]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sixlowpan]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sixlowpan_ctx]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sixlowpan_sfr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sock_util]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-timex]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-tsrb]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-turo]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-uri_parser]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ut_process]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-uuid]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-vfs]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-zptr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ztimer64]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ztimer]	PASSED
✔	01. Task 04 - Unittests on iotlab-m3	PASSED
✔	03. Task 01 - ICMPv6 multicast echo on native	PASSED
✔	03. Task 02 - ICMPv6 link-local echo on native	PASSED
✔	03. Task 03 - ICMPv6 link-local echo on native (1 hour)	PASSED
✔	03. Task 04 - ICMPv6 stress test on native (1 hour)	PASSED
🟡	03. Task 05 - ICMPv6 stress test on native (neighbor cache stress)	SKIPPED
✔	03. Task 06 - ICMPv6 link-local echo on native (IPv6 fragmentation)	PASSED
✔	04. Task 01 - ICMPv6 link-local echo with iotlab-m3	PASSED
✔	04. Task 02 - ICMPv6 multicast echo with iotlab-m3/samr21-xpro	PASSED
✔	04. Task 03 - ICMPv6 echo with large payload	PASSED
✔	04. Task 04 - ICMPv6 echo with iotlab-m3/samr21-xpro 15 minutes	PASSED
🟡	04. Task 05 (Experimental) - ICMPv6 multicast echo with samr21-xpro/cc2538 [lille]	SKIPPED
🟡	04. Task 06 (Experimental)- ICMPv6 link-local echo with samr21-xpro/cc2538 [lille]	SKIPPED
🟡	04. Task 07 (Experimental) - ICMPv6 multicast echo with samr21-xpro/zero + xbee	SKIPPED
🟡	04. Task 08 (Experimental) - ICMPv6 echo with samr21-xpro/zero + xbee	SKIPPED
✔	04. Task 09 - ICMPv6 stress test on iotlab-m3	PASSED
✔	04. Task 10 (Experimental) - ICMPv6 echo with large payload (IPv6 fragmentation)	PASSED
🟡	04. Task 11 (Experimental) - ICMPv6 stress test on nrf802154	SKIPPED
🟡	04. Task 12 (Experimental) - ICMPv6 multicast echo with iotlab-m3/nrf802154	SKIPPED
🟡	04. Task 13 (Experimental) - ICMPv6 link-local echo with iotlab-m3/nrf802154	SKIPPED
✔	05. Task 01 - ICMPv6 echo unicast addresess on native (default route)	PASSED
✔	05. Task 02 - ICMPv6 echo unicast addresess on iotlab-m3 (default route)	PASSED
✔	05. Task 04 - ICMPv6 echo unicast addresess on iotlab-m3 (static route)	PASSED
✔	06. Task 01 - UDP on iotlab-m3	PASSED
✔	06. Task 02 - UDP on iotlab-m3 (UDP port compression)	PASSED
✔	06. Task 03 - UDP on native (non-existent neighbor)	PASSED
✔	06. Task 04 - UDP on iotlab-m3 (non-existent neighbor)	PASSED
✔	06. Task 05 - Empty UDP on native	PASSED
✔	06. Task 06 - Empty UDP on iotlab-m3	PASSED
✔	07. Task 01 - ICMPv6 echo on iotlab-m3 with three hops (static route)	PASSED
✔	07. Task 02 - UDP on iotlab-m3 with three hops (static route)	PASSED
✔	07. Task 03 - ICMPv6 echo on iotlab-m3 with three hops (RPL route)	PASSED
✔	07. Task 04 - UDP on iotlab-m3 with three hops (RPL route)	PASSED
🟡	07. Task 05 (Experimental) - UDP with large payload on iotlab-m3 with three hops (RPL route)	SKIPPED
✔	08. Task 01 - ICMPv6 echo between native and Linux	PASSED
✔	08. Task 08 - UDP between GNRC and lwIP on iotlab-m3	PASSED
✔	09. Task 01 - CORD Endpoint	PASSED
✔	09. Task 02 - Confirmable retries [False-0]	PASSED
✔	09. Task 02 - Confirmable retries [True-1]	PASSED
✔	09. Task 03 - Block1	PASSED
✔	09. Task 04 - Block2	PASSED
✔	09. Task 05 - Observe registration and notification	PASSED
✔	10. Task 01 - Destination unreachable - no route to destination	PASSED
✔	10. Task 02 - Destination unreachable - Beyond scope of source address	PASSED
✔	10. Task 03 - Destination unreachable - address unreachable (target node address)	PASSED
✔	10. Task 04 - Destination unreachable - address unreachable (neighbor cache miss)	PASSED
✔	10. Task 05 - Destination unreachable - port unreachable	PASSED
✔	10. Task 06 - Destination unreachable - port unreachable (large payload)	PASSED
✔	10. Task 07 - Packet too big	PASSED
✔	10. Task 08 - Time exceeded - hop limit exceeded in transit	PASSED
✔	10. Task 09 - Parameter problem - erroneous header field encountered	PASSED
✔	10. Task 10 - IPv6-in-IPv6 encapsulation	PASSED
✖	11. Task 01 - LoRaWAN example	FAILED
✔	11. Task 02 - OTAA join procedure	PASSED
✔	11. Task 03 - ABP join procedure	PASSED
✔	11. Task 04 - LoRaWAN device parameters persistence	PASSED
✔	11. Task 05 - GNRC LoRaWAN (Over The Air Activation) [otaa]	PASSED
🟡	11. Task 06 - GNRC LoRaWAN (ABP) [abp]	SKIPPED

[fjmolinas]

I think the remaining failing LoRa test is probably due to RIOT-OS/RIOT#11237, could you take a look @aabadie?

[fjmolinas]

Furthermore, I’m running the compile tests and get a bunch of errors for multiple applications on various boards.

@OlegHahm can you give more details?

[OlegHahm]

Furthermore, I’m running the compile tests and get a bunch of errors for multiple applications on various boards.

@OlegHahm can you give more details?

As far as I can see most problems did either occur because of missing toolchains on host A or too modern compiler on host B. As soon as the LoRa issues are sorted out, I will run the buildtest in the CI.

[miri64]

As far as I can see most problems did either occur because of missing toolchains on host A or too modern compiler on host B.

Hint: if you are using the compile_and_test_for_board.py script, you can use the --report-xml flag to output in junit XML format (given you have the junit_xml packet installed). With the junit2html tool you then can generate nicely formatted HTML, similar to the one that can be found in the artifacts of the release-tests.

[OlegHahm]

Currently I'm waiting for RIOT-OS/RIOT#17974 to be reviewed, merged, and backported. As soon as @jia200x can confirm that this fixes the LoRa issues, I'm ready for RC4.

[OlegHahm]

Closing in favor of #245