Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests/periph_hwrng: add automated python test #11448

Merged
merged 2 commits into from
Apr 30, 2019
Merged

tests/periph_hwrng: add automated python test #11448

merged 2 commits into from
Apr 30, 2019

Conversation

cladmi
Copy link
Contributor

@cladmi cladmi commented Apr 25, 2019 •
edited
Loading

Contribution description

Add the automated test for 'periph_hwrng'. It only checks that numbers
are returned, not their value.

I did not enable 'CI: run tests' as none of the CI boards have 'hwrng' and native tests are run anyway.

Testing procedure

Run the test on one of the supported boards:

acd52832 airfy-beacon arduino-due b-l072z-lrwan1 b-l475e-iot01a calliope-mini cc2538dk esp32-mh-et-live-minikit esp32-olimex-evb esp32-wemos-lolin-d32-pro esp32-wroom-32 esp32-wrover-kit esp8266-esp-12x esp8266-olimex-mod esp8266-sparkfun-thing f4vi1 firefly frdm-k22f frdm-k64f microbit msbiot mulle native nrf51dk nrf51dongle nrf52832-mdk nrf52840-mdk nrf52840dk nrf52dk nrf6310 nucleo-f207zg nucleo-f410rb nucleo-f412zg nucleo-f413zh nucleo-f429zi nucleo-f722ze nucleo-f746zg nucleo-f767zi nucleo-l053r8 nucleo-l073rz nucleo-l432kc nucleo-l433rc nucleo-l452re nucleo-l476rg nucleo-l496zg openmote-b openmote-cc2538 pba-d-01-kw2x pic32-wifire
pyboard reel remote-pa remote-reva remote-revb ruuvitag stm32f429i-disc1 stm32f4discovery stm32f769i-disco stm32l476g-disco thingy52 ublox-c030-u201 udoo yunjia-nrf51822

Successfull test:

Expected output

BOARD=board make -C tests/periph_hwrng flash test 

2019-04-25 15:30:41,510 - INFO # generating 11 random byte(s)
2019-04-25 15:30:41,516 - INFO # Got: 0x5c 0xa3 0x12 0x8a 0x33 0xc6 0xd3 0x8f 0x48 0xeb 0x03
2019-04-25 15:30:41,518 - INFO # generating 12 random byte(s)
2019-04-25 15:30:41,524 - INFO # Got: 0x71 0x09 0xc4 0x91 0xcf 0xb1 0x32 0x49 0x76 0x3a 0xb1 0xca
2019-04-25 15:30:41,527 - INFO # generating 13 random byte(s)
2019-04-25 15:30:41,533 - INFO # Got: 0xc4 0xc0 0xeb 0x7b 0x66 0xe1 0xd3 0x09 0x3a 0x5b 0xcf 0x23 0x67
2019-04-25 15:30:41,535 - INFO # generating 14 random byte(s)
2019-04-25 15:30:41,542 - INFO # Got: 0xc4 0x14 0x05 0xd7 0x46 0x33 0x20 0xbe 0xf9 0xe7 0x80 0xab 0xb3 0x73
2019-04-25 15:30:41,545 - INFO # generating 15 random byte(s)
2019-04-25 15:30:41,552 - INFO # Got: 0x1b 0xae 0x21 0x68 0xbb 0x66 0x4d 0xed 0x0f 0xe1 0xeb 0x04 0x0a 0x3b 0x60
2019-04-25 15:30:41,554 - INFO # generating 16 random byte(s)
2019-04-25 15:30:41,562 - INFO # Got: 0x63 0x98 0x85 0x9b 0x2e 0x10 0xdc 0x78 0x3e 0x20 0x87 0x40 0xc9 0xb4 0xd1 0xa0
2019-04-25 15:30:41,565 - INFO # generating 17 random byte(s)
2019-04-25 15:30:41,573 - INFO # Got: 0x84 0x44 0x8e 0xbb 0x4a 0x83 0x94 0x78 0x52 0x82 0xde 0x44 0x6b 0xb9 0xae 0xaf 0xa4
2019-04-25 15:30:41,575 - INFO # generating 18 random byte(s)
2019-04-25 15:30:41,584 - INFO # Got: 0x65 0xa9 0x8e 0x5b 0xba 0x2f 0x63 0xe1 0x9e 0xf4 0x54 0xb0 0x9d 0x4b 0x8f 0x52 0xeb 0x12
2019-04-25 15:30:41,586 - INFO # generating 19 random byte(s)
2019-04-25 15:30:41,595 - INFO # Got: 0x4f 0x82 0xf8 0x77 0x59 0x3e 0x0c 0x54 0xa1 0x4b 0xdc 0x5f 0x42 0xd7 0xe6 0x2d 0x03 0x9d 0xa5
2019-04-25 15:30:41,598 - INFO # generating 20 random byte(s)
2019-04-25 15:30:41,607 - INFO # Got: 0xb1 0xdf 0x89 0x45 0xd3 0xda 0xd7 0xdc 0xdf 0x9b 0xba 0x20 0x97 0xa1 0x0f 0xf3 0x32 0x1c 0x0f 0xaa
2019-04-25 15:30:42,053 - INFO # main(): This is RIOT! (Version: 2019.07-devel-131-g9a15b-pr/tests/periph_hwrng)
2019-04-25 15:30:42,053 - INFO #
2019-04-25 15:30:42,056 - INFO # HWRNG peripheral driver test
2019-04-25 15:30:42,056 - INFO #
2019-04-25 15:30:42,061 - INFO # This test will print from 1 to 20 random bytes about every second
2019-04-25 15:30:42,061 - INFO #
2019-04-25 15:30:42,064 - INFO # generating 1 random byte(s)
2019-04-25 15:30:42,065 - INFO # Got: 0xaf
2019-04-25 15:30:42,067 - INFO # generating 2 random byte(s)
2019-04-25 15:30:42,068 - INFO # Got: 0xa4 0x76
2019-04-25 15:30:42,071 - INFO # generating 3 random byte(s)
2019-04-25 15:30:42,073 - INFO # Got: 0x07 0xb2 0xe4
2019-04-25 15:30:42,075 - INFO # generating 4 random byte(s)
2019-04-25 15:30:42,077 - INFO # Got: 0x87 0x6a 0x5b 0x5d
2019-04-25 15:30:42,080 - INFO # generating 5 random byte(s)
2019-04-25 15:30:42,082 - INFO # Got: 0xe5 0x1a 0x1f 0x29 0xe2
2019-04-25 15:30:42,085 - INFO # generating 6 random byte(s)
2019-04-25 15:30:42,088 - INFO # Got: 0x4a 0xf4 0x7c 0xfd 0xbe 0x5c
2019-04-25 15:30:42,090 - INFO # generating 7 random byte(s)
2019-04-25 15:30:42,094 - INFO # Got: 0xd3 0xff 0x1c 0xdd 0x5e 0x33 0x69
2019-04-25 15:30:42,096 - INFO # generating 8 random byte(s)
2019-04-25 15:30:42,100 - INFO # Got: 0x32 0xbe 0x08 0x63 0x9e 0x51 0x9e 0x60
2019-04-25 15:30:42,103 - INFO # generating 9 random byte(s)
2019-04-25 15:30:42,107 - INFO # Got: 0xd0 0x96 0xba 0x0d 0xb7 0x59 0xd7 0xee 0x88
2019-04-25 15:30:42,109 - INFO # generating 10 random byte(s)
2019-04-25 15:30:42,114 - INFO # Got: 0x39 0xc4 0x62 0x08 0xe6 0x01 0xfa 0xb6 0x71 0x96
2019-04-25 15:30:42,117 - INFO # generating 11 random byte(s)
2019-04-25 15:30:42,122 - INFO # Got: 0x20 0x6d 0x3f 0x63 0x84 0x59 0xee 0x2f 0x1a 0x85 0x41
2019-04-25 15:30:42,125 - INFO # generating 12 random byte(s)
2019-04-25 15:30:42,131 - INFO # Got: 0xa1 0x5c 0xf4 0x78 0x43 0xfa 0xd2 0x9c 0x87 0x96 0x6a 0x12
2019-04-25 15:30:42,133 - INFO # generating 13 random byte(s)
2019-04-25 15:30:42,139 - INFO # Got: 0xfe 0xf8 0x43 0x44 0x08 0x60 0x2b 0x58 0xd8 0xfa 0x1b 0x8b 0x37
2019-04-25 15:30:42,142 - INFO # generating 14 random byte(s)
2019-04-25 15:30:42,149 - INFO # Got: 0x63 0x9f 0xdb 0x7e 0x62 0x5b 0x85 0x6e 0x0f 0xf8 0x3b 0x28 0xb8 0xda
2019-04-25 15:30:42,151 - INFO # generating 15 random byte(s)
2019-04-25 15:30:42,158 - INFO # Got: 0x0e 0x27 0x33 0xee 0x39 0x8d 0xf2 0x05 0x1b 0x26 0x06 0xdd 0xf4 0x43 0x25
2019-04-25 15:30:42,161 - INFO # generating 16 random byte(s)
2019-04-25 15:30:42,168 - INFO # Got: 0xf9 0x4b 0xea 0x9e 0x88 0xc8 0xcb 0xb5 0x9b 0x06 0x13 0xc2 0x76 0xb5 0x65 0xa1
2019-04-25 15:30:42,171 - INFO # generating 17 random byte(s)
2019-04-25 15:30:42,179 - INFO # Got: 0x2c 0xd5 0x40 0xc2 0xec 0x8c 0x8d 0x91 0x95 0x75 0x6c 0xa9 0x2c 0xca 0x0f 0xb1 0x88
2019-04-25 15:30:42,182 - INFO # generating 18 random byte(s)
2019-04-25 15:30:42,190 - INFO # Got: 0xe0 0x85 0xce 0x2c 0xf5 0xec 0x49 0x27 0x2c 0x24 0x4f 0x1e 0xc4 0x21 0x0f 0x29 0xea 0x0d
2019-04-25 15:30:42,193 - INFO # generating 19 random byte(s)
2019-04-25 15:30:42,202 - INFO # Got: 0xb4 0x3e 0x4d 0xed 0xdd 0xa4 0x02 0xee 0x8e 0xa3 0xfb 0xac 0x80 0x32 0x93 0x68 0x62 0xc7 0x15
2019-04-25 15:30:42,204 - INFO # generating 20 random byte(s)
2019-04-25 15:30:42,214 - INFO # Got: 0xdc 0xc2 0x28 0x89 0x13 0xad 0xe6 0x45 0xd9 0xb4 0x4d 0x0c 0xbd 0x83 0x01 0xb6 0x35 0x84 0xb2 0xbd

make: Leaving directory '/home/harter/work/git/worktree/riot_master/tests/periph_hwrng'

Failure outputBOARD=frdm-k64f make -C tests/periph_hwrng flash test 

019-04-25 15:33:47,094 - INFO # main(): This is RIOT! (Version: buildtest)
2019-04-25 15:33:47,094 - INFO #
2019-04-25 15:33:47,097 - INFO # HWRNG peripheral driver test
2019-04-25 15:33:47,097 - INFO #
2019-04-25 15:33:47,103 - INFO # This test will print from 1 to 20 random bytes about every second
2019-04-25 15:33:47,103 - INFO #
2019-04-25 15:33:47,105 - INFO # generating 1 random byte(s)
2019-04-25 15:33:47,105 - INFO #
2019-04-25 15:33:47,108 - INFO # Context before hardfault:
2019-04-25 15:33:47,109 - INFO #    r0: 0x1fff0a44
2019-04-25 15:33:47,111 - INFO #    r1: 0x00000001
2019-04-25 15:33:47,112 - INFO #    r2: 0x1fff0ce4
2019-04-25 15:33:47,114 - INFO #    r3: 0x40029000
2019-04-25 15:33:47,115 - INFO #   r12: 0x1fff09db
2019-04-25 15:33:47,117 - INFO #    lr: 0x000018fd
2019-04-25 15:33:47,118 - INFO #    pc: 0x00001a46
2019-04-25 15:33:47,120 - INFO #   psr: 0x21000200
2019-04-25 15:33:47,120 - INFO #
2019-04-25 15:33:47,121 - INFO # FSR/FAR:
2019-04-25 15:33:47,122 - INFO #  CFSR: 0x00008600
2019-04-25 15:33:47,124 - INFO #  HFSR: 0x40000000
2019-04-25 15:33:47,126 - INFO #  DFSR: 0x00000000
2019-04-25 15:33:47,127 - INFO #  AFSR: 0x00000000
2019-04-25 15:33:47,129 - INFO #  BFAR: 0x40029004
2019-04-25 15:33:47,129 - INFO # Misc
2019-04-25 15:33:47,131 - INFO # EXC_RET: 0xfffffffd
2019-04-25 15:33:47,135 - INFO # Attempting to reconstruct state for debugging...
2019-04-25 15:33:47,136 - INFO # In GDB:
2019-04-25 15:33:47,137 - INFO #   set $pc=0x1a46
2019-04-25 15:33:47,138 - INFO #   frame 0
2019-04-25 15:33:47,139 - INFO #   bt
2019-04-25 15:33:47,139 - INFO #
2019-04-25 15:33:47,142 - INFO # ISR stack overflowed by at least 16 bytes.
Timeout in expect script at "child.expect(generated)" (tests/periph_hwrng/tests/01-run.py:33)

Issues/PRs references

Written while working on: #11447 running release tests

@cladmi
tests/periph_hwrng: add automated python test
9a15b32 
Add the automated test for 'periph_hwrng'. It only checks that numbers
are returned, not their value.
@cladmi cladmi added Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Apr 25, 2019
@cladmi cladmi added this to the Release 2019.04 milestone Apr 25, 2019
@cladmi cladmi mentioned this pull request Apr 26, 2019
Merged
@fjmolinas fjmolinas self-requested a review April 30, 2019 07:02
fjmolinas
fjmolinas approved these changes Apr 30, 2019
View reviewed changes
Copy link
Contributor

@fjmolinas fjmolinas left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cladmi tested and passing for:

  • nucleo-l073rg
  • nucleo-l476rg
  • pba-d-01-kw2x

Although I would like all the boards to be tested, I think this PR aims at adding a basic test. If some boards don't pass the test the problem would be with that board/cpu implementation. So ACK

NOTE: I think a porpper test on these modules will mean implementing (NIST) Statistical Test Suite (STS) or something like that, but then maybe not all hwrng pass those test suites, not because they are not properly implemented but because they aren't random enough.

@cladmi
Copy link
Contributor Author

cladmi commented Apr 30, 2019

I do not know why, but I missed that the nrf52dk has hwrng.
Also, I forgot it needs to be enabled to run in murdock even native. I will push this and wait for murdock output.

@cladmi cladmi added the CI: run tests If set, CI server will run tests on hardware for the labeled PR label Apr 30, 2019
@cladmi
Copy link
Contributor Author

cladmi commented Apr 30, 2019

@fjmolinas Agreed, no need to run the test on all boards, it was to at least try a few of them and the test may be adapted later in case we find an issue with one specific board.

NOTE: I think a porpper test on these modules will mean implementing (NIST) Statistical Test Suite (STS) or something like that, but then maybe not all hwrng pass those test suites, not because they are not properly implemented but because they aren't random enough.

For the frdm-k64f as I was reading the datasheet, it says that only one bit of entropy should be taken out per random read where we extract 32 right now. So indeed having a way of testing this could be nice.

It could be more in a specific test like periph_hwrng_rng_quality as there is a difference between having the hardware answering, and giving "correct" output.

@cladmi
Copy link
Contributor Author

cladmi commented Apr 30, 2019

The test output for native and nrf52dk correctly match what is expected.

@fjmolinas I let you hit merge if you accept my last change.

@fjmolinas
Copy link
Contributor

@cladmi all good go!

@fjmolinas fjmolinas merged commit c073e56 into RIOT-OS:master Apr 30, 2019
@cladmi
Copy link
Contributor Author

cladmi commented Apr 30, 2019

Thank you for the review.

@cladmi cladmi deleted the pr/tests/periph_hwrng branch April 30, 2019 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fjmolinas fjmolinas fjmolinas approved these changes

Assignees
No one assigned
Labels
Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: run tests If set, CI server will run tests on hardware for the labeled PR
Projects
None yet
Milestone
Release 2019.04
Development

Successfully merging this pull request may close these issues.
2 participants
@cladmi @fjmolinas