Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release 2019.04 - RC2 #124

Closed
danpetry opened this issue Apr 24, 2019 · 14 comments
Closed

Release 2019.04 - RC2 #124

danpetry opened this issue Apr 24, 2019 · 14 comments
Labels
help wanted

Comments

@danpetry
Copy link
Contributor

danpetry commented Apr 24, 2019 •
edited
Loading

This issue is for discussion related to testing and bugfixing of Release Candidate 2 of the 2019.04 release.

To track testing, please use the tracking spreadsheet, which contains the list of tests and a column for your initials if you're testing something, and pass/fail status for your test.

https://docs.google.com/spreadsheets/d/1ytFlAysdAUissgenypXMErTuIHR1mfgy09m0LTQpu_o/edit?usp=sharing

There is a "test failures and bugfixes" tab in the spreadsheet, which is there to capture the progress of bugfixes coming out of the tests. This is optional for you, as I'll keep it updated, but hopefully will be also useful.

There are also folders for you to drop test artefacts in, if there are any:

https://drive.google.com/drive/folders/10Dmibbnpv4giDbT2fzBVGSOf2VIzD_8t?usp=sharing
@danpetry danpetry mentioned this issue Apr 24, 2019
Closed
@danpetry
Copy link
Contributor Author

I've marked all the tests which I think don't need to be run as N/A. The reason why I believe they don't need to be run is that they passed the first time, and none of the PRs backported to the release branch would affect whether they pass or not. Please let me know if you disagree with this, or can see any tests marked N/A which don't fit these criteria.

@miri64
Copy link
Member

miri64 commented Apr 24, 2019 •
edited
Loading

I agree with the skipping of most of the GNRC tests. Only changes to GNRC itself were the IPv6 address auto-configuration for link-layers without addresses (RIOT-OS/RIOT#11430), which are not covered by the test, however we should at least test if address auto-configuration with link-layer addresses still works (so 3.1 and 4.1 make sense to me).

Likewise, the UDP test was fixed for 0-length payloads (RIOT-OS/RIOT#11357), so we should at least redo 6.5 and 6.6 and either 6.1 or 6.2 (so also agreed with 6.1, 6.5, and 6.6 to be retested).

$ git log --stat --oneline --no-merges 2019.04-RC1..2019.04-RC2 | grep -i -C1 gnrc
60ab0a9 gnrc_ipv6_nib: don't autoconfig IPv6 address without L2 addr
 sys/net/gnrc/network_layer/ipv6/nib/_nib-slaac.c | 4 ++++
 1 file changed, 4 insertions(+)
--
 1 file changed, 4 insertions(+), 3 deletions(-)
e48bfb5 gnrc_netif_ieee802154: check if mhr_len is lesser than nread
 sys/net/gnrc/netif/ieee802154/gnrc_netif_ieee802154.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
7fe868e tests/gnrc_udp: fix printf format string (%d->%u).
 tests/gnrc_udp/udp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
2247383 tests/gnrc_udp: Replace atoi() by strtol().
 tests/gnrc_udp/udp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
480dfcb Give time to allow node to set its ll address
 tests/gnrc_sock_dns/tests/01-run.py | 2 ++
 1 file changed, 2 insertions(+)

I don't see the need to retest 4.3, 5.2, and 5.4 though.

@miri64
Copy link
Member

miri64 commented Apr 24, 2019

I agree with the skipping of most of the GNRC tests. Only changes to GNRC itself were the IPv6 address auto-configuration for link-layers without addresses (RIOT-OS/RIOT#11430), which are not covered by the test, however we should at least test if address auto-configuration with link-layer addresses still works (so 3.1 and 4.1 make sense to me).

Actually, we should rather go for 3.2 instead of 3.1 as 3.1 uses multicast addresses.

@danpetry
Copy link
Contributor Author

I don't see the need to retest 4.3, 5.2, and 5.4 though.

We changed the test parameters for those ones to make the timing longer. They should pass, I guess, since they were passing last time with no changes to the code...

@miri64
Copy link
Member

miri64 commented Apr 24, 2019

We changed the test parameters for those ones to make the timing longer. They should pass, I guess, since they were passing last time with no changes to the code...

Got it (when I was doing them ;-))

@miri64
Copy link
Member

miri64 commented Apr 24, 2019

All GNRC-related tests that needed to be re-done pass.

@miri64
Copy link
Member

miri64 commented Apr 24, 2019

Forgot to do 6.1. It's running now.

@danpetry
Copy link
Contributor Author

great thanks

@miri64
Copy link
Member

miri64 commented Apr 24, 2019

Okay, 6.1 retested (also rechecked packet buffer emptiness). All are passing :-).

@cladmi
Copy link
Contributor

cladmi commented Apr 26, 2019

I finally start to have results of running tests on boards. I currently have only RC1 though.

It would be great to have RIOT-OS/RIOT#11454 for the release.

All the following tests are failing since this release for the frdm-k64f because of the hwrng being enabled by default while not working on the board. Also examples/gnrc_networking does not work because of it.

Otherwise it would be necessary to mention it.

--- 2019.04-RC1_periph_hwrng_fix/frdm-k64f/failuresummary.md    2019-04-26 16:50:14.000000000 +0200
+++ 2019.04-RC1/frdm-k64f/failuresummary.md     2019-04-24 19:17:40.000000000 +0200
@@ -1,8 +1,29 @@
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
+- [tests/gnrc_ipv6_nib](tests/gnrc_ipv6_nib/test.failed)
+- [tests/gnrc_ipv6_nib_6ln](tests/gnrc_ipv6_nib_6ln/test.failed)
+- [tests/gnrc_ndp](tests/gnrc_ndp/test.failed)
+- [tests/gnrc_netif](tests/gnrc_netif/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sixlowpan](tests/gnrc_sixlowpan/test.failed)
 - [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/gnrc_sock_ip](tests/gnrc_sock_ip/test.failed)
+- [tests/gnrc_sock_udp](tests/gnrc_sock_udp/test.failed)
+- [tests/nhdp](tests/nhdp/test.failed)
 - [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/periph_timer](tests/periph_timer/test.failed)
+- [tests/pkg_c25519](tests/pkg_c25519/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
+- [tests/pkg_hacl](tests/pkg_hacl/test.failed)
+- [tests/pkg_libcoap](tests/pkg_libcoap/test.failed)
+- [tests/pkg_libcose](tests/pkg_libcose/test.failed)
+- [tests/pkg_libhydrogen](tests/pkg_libhydrogen/test.failed)
+- [tests/pkg_micro-ecc-with-hwrng](tests/pkg_micro-ecc-with-hwrng/test.failed)
+- [tests/pkg_monocypher](tests/pkg_monocypher/test.failed)
+- [tests/pkg_qdsa](tests/pkg_qdsa/test.failed)
+- [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed)
 - [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)
+- [tests/pthread_barrier](tests/pthread_barrier/test.failed)
+- [tests/pthread_rwlock](tests/pthread_rwlock/test.failed)
+- [tests/rng](tests/rng/test.failed)
+- [tests/trickle](tests/trickle/test.failed)

@cladmi
Copy link
Contributor

cladmi commented Apr 26, 2019

Tests results for the 2019.04-RC1 compared to the previous release.

I need to use the commits from RIOT-OS/RIOT#10870 to be able to run the tests on my setup. It looks like a lot of improvements for the arduino-mega2560 but did not invesigate why yet.

As it was tested with -RC1 the periph_rtt fix was not in there.

--- 2019.01/failuresummary.md	2019-04-26 15:45:07.739741954 +0200
+++ 2019.04-RC1/failuresummary.md	2019-04-26 16:06:40.000000000 +0200
@@ -1,144 +1,155 @@
 #### arduino-mega2560/failuresummary.md
 
 Failures during test:
 - [tests/bitarithm_timings](tests/bitarithm_timings/test.failed)
-- [tests/cb_mux](tests/cb_mux/test.failed)
-- [tests/cb_mux_bench](tests/cb_mux_bench/test.failed)
-- [tests/eepreg](tests/eepreg/test.failed)
 - [tests/evtimer_msg](tests/evtimer_msg/test.failed)
-- [tests/gnrc_sock_ip](tests/gnrc_sock_ip/test.failed)
 - [tests/isr_yield_higher](tests/isr_yield_higher/test.failed)
 - [tests/libfixmath](tests/libfixmath/test.failed)
-- [tests/mutex_order](tests/mutex_order/test.failed)
-- [tests/netdev_test](tests/netdev_test/test.failed)
-- [tests/od](tests/od/test.failed)
 - [tests/periph_gpio](tests/periph_gpio/test.failed)
-- [tests/periph_timer](tests/periph_timer/test.failed)
 - [tests/pipe](tests/pipe/test.failed)
-- [tests/pkg_cayenne-lpp](tests/pkg_cayenne-lpp/test.failed)
+- [tests/pkg_c25519](tests/pkg_c25519/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
 - [tests/pkg_jsmn](tests/pkg_jsmn/test.failed)
 - [tests/pkg_libb2](tests/pkg_libb2/test.failed)
 - [tests/pkg_lora-serialization](tests/pkg_lora-serialization/test.failed)
 - [tests/pkg_micro-ecc](tests/pkg_micro-ecc/test.failed)
 - [tests/pkg_tiny-asn1](tests/pkg_tiny-asn1/test.failed)
-- [tests/pkg_u8g2](tests/pkg_u8g2/test.failed)
 - [tests/posix_semaphore](tests/posix_semaphore/test.failed)
 - [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)
-- [tests/riotboot_hdr](tests/riotboot_hdr/test.failed)
-- [tests/rmutex](tests/rmutex/test.failed)
-- [tests/rng](tests/rng/test.failed)
-- [tests/sched_testing](tests/sched_testing/test.failed)
-- [tests/shell](tests/shell/test.failed)
-- [tests/thread_flags](tests/thread_flags/test.failed)
-- [tests/thread_flags_xtimer](tests/thread_flags_xtimer/test.failed)
-- [tests/thread_msg](tests/thread_msg/test.failed)
-- [tests/thread_msg_seq](tests/thread_msg_seq/test.failed)
-- [tests/thread_race](tests/thread_race/test.failed)
 - [tests/trickle](tests/trickle/test.failed)
-- [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.failed)
-- [tests/xtimer_reset](tests/xtimer_reset/test.failed)
 
 #### frdm-k64f/failuresummary.md
 
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
+- [tests/gnrc_ipv6_nib](tests/gnrc_ipv6_nib/test.failed)
+- [tests/gnrc_ipv6_nib_6ln](tests/gnrc_ipv6_nib_6ln/test.failed)
+- [tests/gnrc_ndp](tests/gnrc_ndp/test.failed)
+- [tests/gnrc_netif](tests/gnrc_netif/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sixlowpan](tests/gnrc_sixlowpan/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/gnrc_sock_ip](tests/gnrc_sock_ip/test.failed)
+- [tests/gnrc_sock_udp](tests/gnrc_sock_udp/test.failed)
+- [tests/nhdp](tests/nhdp/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/periph_timer](tests/periph_timer/test.failed)
+- [tests/pkg_c25519](tests/pkg_c25519/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
+- [tests/pkg_hacl](tests/pkg_hacl/test.failed)
+- [tests/pkg_libcoap](tests/pkg_libcoap/test.failed)
+- [tests/pkg_libcose](tests/pkg_libcose/test.failed)
+- [tests/pkg_libhydrogen](tests/pkg_libhydrogen/test.failed)
 - [tests/pkg_micro-ecc-with-hwrng](tests/pkg_micro-ecc-with-hwrng/test.failed)
-- [tests/pkg_umorse](tests/pkg_umorse/test.failed)
+- [tests/pkg_monocypher](tests/pkg_monocypher/test.failed)
+- [tests/pkg_qdsa](tests/pkg_qdsa/test.failed)
+- [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed)
 - [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)
+- [tests/pthread_barrier](tests/pthread_barrier/test.failed)
+- [tests/pthread_rwlock](tests/pthread_rwlock/test.failed)
+- [tests/rng](tests/rng/test.failed)
+- [tests/trickle](tests/trickle/test.failed)
 
 #### frdm-kw41z/failuresummary.md
 
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
-- [tests/lwip_sock_ip](tests/lwip_sock_ip/test.failed)
-- [tests/lwip_sock_udp](tests/lwip_sock_udp/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/periph_timer](tests/periph_timer/test.failed)
+- [tests/pkg_c25519](tests/pkg_c25519/test.failed)
 - [tests/posix_semaphore](tests/posix_semaphore/test.failed)
 - [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)
 - [tests/thread_flags](tests/thread_flags/test.failed)
 - [tests/unittests](tests/unittests/test.failed)
 - [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.failed)
 - [tests/xtimer_usleep](tests/xtimer_usleep/test.failed)
 
 #### msba2/failuresummary.md
 
 Failures during test:
 - [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.failed)
 - [tests/driver_my9221](tests/driver_my9221/test.failed)
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sixlowpan](tests/gnrc_sixlowpan/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
 - [tests/libfixmath](tests/libfixmath/test.failed)
 - [tests/libfixmath_unittests](tests/libfixmath_unittests/test.failed)
 - [tests/lwip_sock_tcp](tests/lwip_sock_tcp/test.failed)
 - [tests/periph_rtc](tests/periph_rtc/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
 - [tests/pkg_libcose](tests/pkg_libcose/test.failed)
 - [tests/pkg_qdsa](tests/pkg_qdsa/test.failed)
 - [tests/pkg_relic](tests/pkg_relic/test.failed)
 - [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed)
 - [tests/pthread_tls](tests/pthread_tls/test.failed)
-- [tests/rng](tests/rng/test.failed)
 - [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.failed)
 
 #### mulle/failuresummary.md
 
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/periph_timer](tests/periph_timer/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
 - [tests/shell](tests/shell/test.failed)
 - [tests/unittests](tests/unittests/test.failed)
 
 #### nrf52dk/failuresummary.md
 
 Failures during compilation:
 - [tests/mcuboot](tests/mcuboot/compilation.failed)
 
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
 
 #### nucleo-f103rb/failuresummary.md
 
 Failures during test:
 - [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.failed)
 - [tests/driver_my9221](tests/driver_my9221/test.failed)
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
 - [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.failed)
 - [tests/xtimer_usleep_short](tests/xtimer_usleep_short/test.failed)
 
 #### pba-d-01-kw2x/failuresummary.md
 
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
 - [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)
 
 #### sltb001a/failuresummary.md
 
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
-- [tests/posix_semaphore](tests/posix_semaphore/test.failed)
 - [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)
 - [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.failed)
 
 #### stm32f3discovery/failuresummary.md
 
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
-- [tests/periph_rtc](tests/periph_rtc/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)

If disabling periph_hwrng for the frdm-k64f the test difference is only:

--- 2019.01/frdm-k64f/failuresummary.md 2019-03-07 18:27:46.000000000 +0100
+++ 2019.04-RC1_periph_hwrng_fix/frdm-k64f/failuresummary.md    2019-04-26 16:50:14.000000000 +0200
@@ -1,8 +1,8 @@
 Failures during test:
 - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
 - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
+- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
+- [tests/periph_rtt](tests/periph_rtt/test.failed)
 - [tests/periph_timer](tests/periph_timer/test.failed)
 - [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)
-- [tests/pkg_micro-ecc-with-hwrng](tests/pkg_micro-ecc-with-hwrng/test.failed)
-- [tests/pkg_umorse](tests/pkg_umorse/test.failed)
 - [tests/ps_schedstatistics](tests/ps_schedstatistics/test.failed)

@cladmi
Copy link
Contributor

cladmi commented Apr 26, 2019 •
edited
Loading

I will run the tests with RC2 during the WE.

@danpetry
Copy link
Contributor Author

As discussed offline, I'm raising a fixup to the release notes to document this failure, by referencing the relevant issue (RIOT-OS/RIOT#11447).

@MrKevinWeiss
Copy link
Contributor

Since we have the release I think we can close...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cladmi @miri64 @danpetry @MrKevinWeiss