Tracking Flaky Tests in CI Pipeline #4246
Comments
|
https://github.com/lf-edge/eve/actions/runs/10820461464?pr=4239 test_suite_pr (amd64, kvm, generic) / Storage test suite (ext4): tests/volume/testdata/volumes_test.txt#L60[stdout]
Reboot Test
=== RUN TestReboot
reboot_test.go:162: time: 2024-09-12T06:21:16.315099837Z out: Wait for state of b23cb68b-686a-4bb1-a9c4-dafe4d101b3b
testContext.go:380: done waiting for State
reboot_test.go:166: time: 2024-09-12T06:21:16.315117117Z out: timewait: 2562047h47m16.854775807s
reboot_test.go:167: time: 2024-09-12T06:21:16.315121566Z out: reboot: false
reboot_test.go:168: time: 2024-09-12T06:21:16.315125644Z out: count: 1
reboot_test.go:172: time: 2024-09-12T06:21:16.315132196Z out: LastRebootTime: 2024-09-12 06:19:16.189524585 +0000 UTC
reboot_test.go:174: time: 2024-09-12T06:21:16.315135993Z out: LastRebootReason: NORMAL: First boot of device - at 2024-09-12T06:19:16.189524585Z
rebooted with reason Reboot reason - system reset, reboot or kernel panic due to watchdog or kernel bug (no kdump) - at 2024-09-12T06:27:53.933490395Z at 2024-09-12 06:27:53.933490395 +0000 UTC/n testProc.go:84: time: 2024-09-12T06:28:37.826564057Z out: projects.ProcInfoFunc done with return: abnormal reboot: Reboot reason - system reset, reboot or kernel panic due to watchdog or kernel bug (no kdump) - at 2024-09-12T06:27:53.933490395Z
testContext.go:271: done for device b23cb68b-686a-4bb1-a9c4-dafe4d101b3b
reboot_test.go:186: time: 2024-09-12T06:28:37.826629611Z out: Number of reboots: 0
test_suite_pr (amd64, kvm, generic) / Storage test suite (ext4): tests/volume/testdata/volumes_test.txt#L60[stdout] Docker app's state test === RUN TestAppStatus apps: '[eclient-mount]' state: 'RUNNING' secs: 900 time: 2024-09-12T06:24:08.38150379Z out: appName eclient-mount state changed to INSTALLED time: 2024-09-12T06:28:37.601723418Z out: appName eclient-mount state changed to UNKNOWN [context canceled] FAIL: ../volume/testdata/volumes_test.txt:60: test interrupted while running command test_suite_pr (amd64, kvm, generic) / LPS LOC test suite: tests/eclient/testdata/profile.txt#L35[stdout]
Reboot Test
=== RUN TestReboot
reboot_test.go:162: time: 2024-09-12T06:21:48.501656078Z out: Wait for state of 29c60d57-2814-4208-8557-58afcde51ddb
testContext.go:380: done waiting for State
reboot_test.go:166: time: 2024-09-12T06:21:48.501686455Z out: timewait: 2562047h47m16.854775807s
reboot_test.go:167: time: 2024-09-12T06:21:48.501696383Z out: reboot: false
reboot_test.go:168: time: 2024-09-12T06:21:48.501706152Z out: count: 1
reboot_test.go:172: time: 2024-09-12T06:21:48.501717142Z out: LastRebootTime: 2024-09-12 06:19:51.701560928 +0000 UTC
reboot_test.go:174: time: 2024-09-12T06:21:48.50172674Z out: LastRebootReason: NORMAL: First boot of device - at 2024-09-12T06:19:51.701560928Z
rebooted with reason Watchdog report for IMGA EVE version 0.0.0-pr4239-ad620da6-kvm-amd64 at 2024-09-12T06:25:43+00:00: 3 /run/zedbox.pid
at 2024-09-12 06:25:44.431647916 +0000 UTC/n testProc.go:84: time: 2024-09-12T06:28:43.389823431Z out: projects.ProcInfoFunc done with return: abnormal reboot: Watchdog report for IMGA EVE version 0.0.0-pr4239-ad620da6-kvm-amd64 at 2024-09-12T06:25:43+00:00: 3 /run/zedbox.pid
test_suite_pr (amd64, kvm, generic) / LPS LOC test suite: tests/eclient/testdata/profile.txt#L35[stdout] Docker app's state test === RUN TestAppStatus apps: '[app-profile-1 app-profile-2 app-profile-1-2 local-manager]' state: 'RUNNING' secs: 1200 time: 2024-09-12T06:23:56.787957058Z out: appName local-manager state changed to RUNNING time: 2024-09-12T06:23:56.787967217Z out: appName app-profile-1 state changed to RESOLVING_TAG time: 2024-09-12T06:23:56.787971276Z out: appName app-profile-2 state changed to RESOLVING_TAG time: 2024-09-12T06:23:56.787974842Z out: appName app-profile-1-2 state changed to UNKNOWN time: 2024-09-12T06:23:58.850038398Z out: appName app-profile-1 state changed to LOADING time: 2024-09-12T06:23:59.850731486Z out: appName app-profile-2 state changed to LOADING time: 2024-09-12T06:24:01.851801418Z out: appName app-profile-1 state changed to CREATING_VOLUME time: 2024-09-12T06:24:02.852097788Z out: appName app-profile-2 state changed to DELIVERED time: 2024-09-12T06:24:02.852127404Z out: appName app-profile-2 state changed to CREATING_VOLUME time: 2024-09-12T06:24:04.853553179Z out: appName app-profile-1-2 state changed to INITIAL time: 2024-09-12T06:24:04.853568234Z out: appName app-profile-1-2 state changed to RESOLVING_TAG time: 2024-09-12T06:24:06.854901Z out: appName app-profile-2 state changed to INSTALLED time: 2024-09-12T06:24:08.856665502Z out: appName app-profile-1 state changed to INSTALLED time: 2024-09-12T06:24:09.857104744Z out: appName app-profile-1-2 state changed to LOADING time: 2024-09-12T06:24:10.857682042Z out: appName app-profile-1-2 state changed to DELIVERED time: 2024-09-12T06:24:12.859356612Z out: appName app-profile-1-2 state changed to CREATING_VOLUME time: 2024-09-12T06:24:14.860979571Z out: appName app-profile-1-2 state changed to INSTALLED [context canceled] FAIL: ../eclient/testdata/profile.txt:35: test interrupted while running command test_suite_pr (amd64, kvm, generic) / User apps test suite: tests/docker/testdata/2dockers_test.txt#L15[stdout]
Reboot Test
=== RUN TestReboot
reboot_test.go:162: time: 2024-09-12T06:24:13.966045332Z out: Wait for state of a85155f4-bef0-4723-a66c-b271bdc1e84f
testContext.go:380: done waiting for State
reboot_test.go:166: time: 2024-09-12T06:24:13.9660643Z out: timewait: 20m0s
reboot_test.go:167: time: 2024-09-12T06:24:13.966067736Z out: reboot: false
reboot_test.go:168: time: 2024-09-12T06:24:13.9660713Z out: count: 2
reboot_test.go:172: time: 2024-09-12T06:24:13.966080921Z out: LastRebootTime: 2024-09-12 06:22:14.793941708 +0000 UTC
reboot_test.go:174: time: 2024-09-12T06:24:13.966095766Z out: LastRebootReason: NORMAL: First boot of device - at 2024-09-12T06:22:14.793941708Z
rebooted with reason Watchdog report for IMGA EVE version 0.0.0-pr4239-ad620da6-kvm-amd64 at 2024-09-12T06:29:00+00:00: 3 /run/zedbox.pid
at 2024-09-12 06:29:01.126397786 +0000 UTC/n testProc.go:84: time: 2024-09-12T06:32:00.839539474Z out: projects.ProcInfoFunc done with return: abnormal reboot: Watchdog report for IMGA EVE version 0.0.0-pr4239-ad620da6-kvm-amd64 at 2024-09-12T06:29:00+00:00: 3 /run/zedbox.pid
test_suite_pr (amd64, kvm, generic) / User apps test suite: tests/docker/testdata/2dockers_test.txt#L15[stdout]
Docker app deployment Test
=== RUN TestDockerStart
docker_test.go:189: time: 2024-09-12T06:27:33.268804656Z out: Add app to list
config changed, to see config run 'eden controller edge-node get-config'
docker_test.go:195: time: 2024-09-12T06:27:34.286300333Z out: Add processing of app messages
docker_test.go:199: time: 2024-09-12T06:27:34.286326971Z out: Add processing of app running messages
docker_test.go:203: time: 2024-09-12T06:27:34.286334506Z out: Add function to obtain EVE IP
docker_test.go:207: time: 2024-09-12T06:27:34.286341178Z out: Add trying to access app via http
testProc.go:84: time: 2024-09-12T06:27:35.133491204Z out: projects.ProcInfoFunc done with return: app found with name t2
testProc.go:84: time: 2024-09-12T06:27:41.881725512Z out: projects.ProcTimerFunc done with return: external ip is: 127.0.0.1
[stderr]
time="2024-09-12T06:27:35Z" level=info msg="time: 2024-09-12T06:27:35.133491204Z out: projects.ProcInfoFunc done with return: app found with name t2"
time="2024-09-12T06:27:41Z" level=info msg="time: 2024-09-12T06:27:41.881725512Z out: projects.ProcTimerFunc done with return: external ip is: 127.0.0.1"
[context canceled]
FAIL: ../docker/testdata/2dockers_test.txt:15: test interrupted while running command
test_suite_pr (amd64, kvm, generic) / Networking test suite: tests/eclient/testdata/nw_switch.txt#L67[stdout]
Reboot Test
=== RUN TestReboot
reboot_test.go:162: time: 2024-09-12T06:25:21.073582414Z out: Wait for state of 0d473afc-e3c3-457b-85a8-5a0416d06329
testContext.go:380: done waiting for State
reboot_test.go:166: time: 2024-09-12T06:25:21.073617129Z out: timewait: 2562047h47m16.854775807s
reboot_test.go:167: time: 2024-09-12T06:25:21.073624062Z out: reboot: false
reboot_test.go:168: time: 2024-09-12T06:25:21.073629902Z out: count: 1
reboot_test.go:172: time: 2024-09-12T06:25:21.073641243Z out: LastRebootTime: 2024-09-12 06:02:26.916189777 +0000 UTC
reboot_test.go:174: time: 2024-09-12T06:25:21.073647105Z out: LastRebootReason: NORMAL: First boot of device - at 2024-09-12T06:02:26.916189777Z
rebooted with reason Watchdog report for IMGA EVE version 0.0.0-pr4239-ad620da6-kvm-amd64 at 2024-09-12T06:33:35+00:00: 3 /run/zedbox.pid
at 2024-09-12 06:33:36.414575825 +0000 UTC/n testProc.go:84: time: 2024-09-12T06:36:30.546873258Z out: projects.ProcInfoFunc done with return: abnormal reboot: Watchdog report for IMGA EVE version 0.0.0-pr4239-ad620da6-kvm-amd64 at 2024-09-12T06:33:35+00:00: 3 /run/zedbox.pid
test_suite_pr (amd64, kvm, generic) / Networking test suite: tests/eclient/testdata/nw_switch.txt#L67[stdout] Docker app's state test === RUN TestAppStatus apps: '[pong]' state: 'RUNNING' secs: 900 time: 2024-09-12T06:32:02.404443833Z out: appName pong state changed to HALTING time: 2024-09-12T06:32:15.37416156Z out: appName pong state changed to HALTED time: 2024-09-12T06:36:30.534785762Z out: appName pong state changed to UNKNOWN [context canceled] FAIL: ../eclient/testdata/nw_switch.txt:67: test interrupted while running command test_suite_pr (amd64, kvm, generic) / Smoke tests (ext4, false): tests/eclient/testdata/shutdown_test.txt#L74[stdout]
Reboot Test
=== RUN TestReboot
reboot_test.go:162: time: 2024-09-12T06:40:13.678565713Z out: Wait for state of 36442791-901e-47c4-974b-3b84aa2f05e2
testContext.go:380: done waiting for State
reboot_test.go:166: time: 2024-09-12T06:40:13.678597244Z out: timewait: 20m0s
reboot_test.go:167: time: 2024-09-12T06:40:13.678604608Z out: reboot: true
reboot_test.go:168: time: 2024-09-12T06:40:13.678609818Z out: count: 1
reboot_test.go:172: time: 2024-09-12T06:40:13.678618916Z out: LastRebootTime: 2024-09-12 06:33:24.682602863 +0000 UTC
reboot_test.go:174: time: 2024-09-12T06:40:13.678624166Z out: LastRebootReason: NORMAL: controller reboot at EVE version 0.0.0-pr4239-ad620da6-kvm-amd64 at 2024-09-12T06:33:24.639407469Z
Example |
|
@OhmSpectator The |
|
Also, we should be careful about what we label as flaky failure. Just because a test fails intermittently it does not necessary mean that the infrastructure is to blame. There could be a race condition inside EVE. |
Agree. That's why I want to track all the failures I see in this ticket and then understand which of them are flaky. |
Thanks for the clarification! A good tip for analysing the results... |
|
Argh... Are the artifacts lost, when running a new attempt? =( |
@milan-zededa, does it look like this? Details
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 2, "major": 6}, "package": "Debian 1:6.2+dfsg-2ubuntu6.22"}, "capabilities": ["oob"]}}
{"return": {}}
{"timestamp": {"seconds": 1726120970, "microseconds": 700472}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/peripheral-anon/device[0]/virtio-backend"}}
{"timestamp": {"seconds": 1726120971, "microseconds": 273852}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/peripheral-anon/device[1]/virtio-backend"}}
{"timestamp": {"seconds": 1726120985, "microseconds": 502002}, "event": "RTC_CHANGE", "data": {"offset": 0}}
{"timestamp": {"seconds": 1726120988, "microseconds": 495149}, "event": "RTC_CHANGE", "data": {"offset": 1}}
{"timestamp": {"seconds": 1726121644, "microseconds": 506808}, "event": "RTC_CHANGE", "data": {"offset": 0}}
{"timestamp": {"seconds": 1726122303, "microseconds": 503429}, "event": "RTC_CHANGE", "data": {"offset": 0}}
{"timestamp": {"seconds": 1726122853, "microseconds": 627509}, "event": "WATCHDOG", "data": {"action": "reset"}}
{"timestamp": {"seconds": 1726122853, "microseconds": 632302}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}
{"timestamp": {"seconds": 1726122854, "microseconds": 467209}, "event": "RTC_CHANGE", "data": {"offset": 0}}
{"timestamp": {"seconds": 1726122854, "microseconds": 572991}, "event": "RTC_CHANGE", "data": {"offset": 0}}
{"timestamp": {"seconds": 1726122991, "microseconds": 504595}, "event": "RTC_CHANGE", "data": {"offset": 0}}
|
Yes, the |
Good to know, thanks! |
Unfortunately, they are. |
|
Interesting. If there is a very clear criterion (#4246 (comment)) for filtering out test failures caused by the watchdog error, could we mark these failures correspondingly in the test result? @yash-zededa, any ideas? |
|
https://github.com/lf-edge/eve/actions/runs/10825096094?pr=4242 test_suite_pr (amd64, kvm, generic) / Smoke tests (ext4, true): tests/eclient/testdata/userdata.txt#L48[stdout] /home/runner/actions-runner/_work/eve/eve/eden/dist/bin/eden sdn fwd eth0 2223 -- ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no -i /home/runner/actions-runner/_work/eve/eve/eden/dist/tests/eclient/image/cert/id_rsa root@FWD_IP -p FWD_PORT grep -q "after_restart" /etc/injected_file.txt Try 1 time="2024-09-12T07:26:05Z" level=fatal msg="command ssh failed: exit status 255" Try 2 time="2024-09-12T07:26:17Z" level=fatal msg="command ssh failed: exit status 255" Try 3 time="2024-09-12T07:26:29Z" level=fatal msg="command ssh failed: exit status 255" Try 4 time="2024-09-12T07:26:41Z" level=fatal msg="command ssh failed: exit status 255" Try 5 [stderr] kex_exchange_identification: read: Connection reset by peer Connection reset by 127.0.0.1 port 2223 Connection timed out during banner exchange Connection to 127.0.0.1 port 2223 timed out Connection timed out during banner exchange Connection to 127.0.0.1 port 2223 timed out Connection timed out during banner exchange Connection to 127.0.0.1 port 2223 timed out [context deadline exceeded] FAIL: ../eclient/testdata/userdata.txt:48: command failure [Valid Issue] test_suite_pr (amd64, kvm, generic) / Storage test suite (zfs): tests/volume/testdata/volumes_test.txt#L40[stdout]
Log/Info/Metric Test
=== RUN TestInfo
lim_test.go:273: time: 2024-09-12T07:30:18.238516525Z out: Wait for info of ebaafcbf-2913-4db8-a7bc-b4433b11a3d0 number=1 timewait=3m0s
test_suite_pr (amd64, kvm, generic) / Smoke tests (zfs, true): tests/eclient/testdata/app_logs.txt#L27[stdout]
Log/Info/Metric Test
=== RUN TestAppLog
lim_test.go:232: time: 2024-09-12T07:35:48.130992136Z out: Wait for app log of ed1a7f8a-efbc-4621-ada8-8f0abab9b37f app eclient number=1 timewait=5m0s
|
|
The failure |
This needs to be handled in the eden. I will create an issue to track this. Depending on the exit status sent by eden/eve commands we can explicitly set the job to fail. |
|
https://github.com/lf-edge/eve/actions/runs/10820422246?pr=4223 [Watchdog] test_suite_pr (amd64, kvm, generic) / Smoke tests (ext4, false): tests/eclient/testdata/app_logs.txt#L27[stdout]
Log/Info/Metric Test
=== RUN TestAppLog
lim_test.go:232: time: 2024-09-12T07:45:03.601893488Z out: Wait for app log of 0ce2f965-340e-424d-98e2-7cf090dc30c8 app eclient number=1 timewait=5m0s
|
|
https://github.com/lf-edge/eve/actions/runs/10825096094?pr=4242 test_suite_pr (amd64, kvm, generic) / Storage test suite (zfs): tests/zfs/testdata/state_and_layout_check.txt#L78[stdout]
Log/Info/Metric Test
=== RUN TestInfo
lim_test.go:273: time: 2024-09-12T12:43:29.071653351Z out: Wait for info of 9eca15d4-45a1-4fd6-a09d-8cc101acf2c5 number=1 timewait=5m0s
|
|
https://github.com/lf-edge/eve/actions/runs/10820422246?pr=4223 test_suite_pr (amd64, kvm, generic) / Smoke tests (ext4, false): tests/eclient/testdata/app_logs.txt#L27[stdout]
Log/Info/Metric Test
=== RUN TestAppLog
lim_test.go:232: time: 2024-09-12T07:45:03.601893488Z out: Wait for app log of 0ce2f965-340e-424d-98e2-7cf090dc30c8 app eclient number=1 timewait=5m0s
|
|
https://github.com/lf-edge/eve/actions/runs/10845977032?pr=4243 [Watchdog] test_suite_pr (amd64, kvm) / Storage test suite (zfs): tests/workflow/testdata/eden_start.txt#L1[stdout]
{"status":"Pulling from library/redis","id":"7"}
{"status":"Pulling fs layer","progressDetail":{},"id":"a2318d6c47ec"}
{"status":"Pulling fs layer","progressDetail":{},"id":"ed7fd66f27f2"}
{"status":"Pulling fs layer","progressDetail":{},"id":"410a3d5b3155"}
{"status":"Pulling fs layer","progressDetail":{},"id":"9312cf3f6b3e"}
{"status":"Pulling fs layer","progressDetail":{},"id":"c39877ab23d0"}
{"status":"Pulling fs layer","progressDetail":{},"id":"01394ffc7248"}
{"status":"Pulling fs layer","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Pulling fs layer","progressDetail":{},"id":"5a03cb6163ab"}
{"status":"Waiting","progressDetail":{},"id":"01394ffc7248"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"9312cf3f6b3e"}
{"status":"Waiting","progressDetail":{},"id":"5a03cb6163ab"}
{"status":"Waiting","progressDetail":{},"id":"c39877ab23d0"}
{"status":"Downloading","progressDetail":{"current":721,"total":874},"progress":"[=========================================\u003e ] 721B/874B","id":"410a3d5b3155"}
{"status":"Downloading","progressDetail":{"current":874,"total":874},"progress":"[==================================================\u003e] 874B/874B","id":"410a3d5b3155"}
{"status":"Verifying Checksum","progressDetail":{},"id":"410a3d5b3155"}
{"status":"Download complete","progressDetail":{},"id":"410a3d5b3155"}
{"status":"Downloading","progressDetail":{"current":720,"total":1098},"progress":"[================================\u003e ] 720B/1.098kB","id":"ed7fd66f27f2"}
{"status":"Downloading","progressDetail":{"current":1098,"total":1098},"progress":"[==================================================\u003e] 1.098kB/1.098kB","id":"ed7fd66f27f2"}
{"status":"Verifying Checksum","progressDetail":{},"id":"ed7fd66f27f2"}
{"status":"Download complete","progressDetail":{},"id":"ed7fd66f27f2"}
{"status":"Downloading","progressDetail":{"current":298232,"total":29126484},"progress":"[\u003e ] 298.2kB/29.13MB","id":"a2318d6c47ec"}
{"status":"Downloading","progressDetail":{"current":8947949,"total":29126484},"progress":"[===============\u003e ] 8.948MB/29.13MB","id":"a2318d6c47ec"}
{"status":"Downloading","progressDetail":{"current":18184429,"total":29126484},"progress":"[===============================\u003e ] 18.18MB/29.13MB","id":"a2318d6c47ec"}
{"status":"Downloading","progressDetail":{"current":26851565,"total":29126484},"progress":"[==============================================\u003e ] 26.85MB/29.13MB","id":"a2318d6c47ec"}
{"status":"Verifying Checksum","progressDetail":{},"id":"a2318d6c47ec"}
{"status":"Download complete","progressDetail":{},"id":"a2318d6c47ec"}
{"status":"Extracting","progressDetail":{"current":294912,"total":29126484},"progress":"[\u003e ] 294.9kB/29.13MB","id":"a2318d6c47ec"}
{"status":"Downloading","progressDetail":{"current":15610,"total":1437818},"progress":"[\u003e ] 15.61kB/1.438MB","id":"9312cf3f6b3e"}
{"status":"Verifying Checksum","progressDetail":{},"id":"9312cf3f6b3e"}
{"status":"Download complete","progressDetail":{},"id":"9312cf3f6b3e"}
{"status":"Downloading","progressDetail":{"current":155703,"total":15316903},"progress":"[\u003e ] 155.7kB/15.32MB","id":"c39877ab23d0"}
{"status":"Extracting","progressDetail":{"current":7372800,"total":29126484},"progress":"[============\u003e ] 7.373MB/29.13MB","id":"a2318d6c47ec"}
{"status":"Downloading","progressDetail":{"current":11188489,"total":15316903},"progress":"[====================================\u003e ] 11.19MB/15.32MB","id":"c39877ab23d0"}
{"status":"Verifying Checksum","progressDetail":{},"id":"c39877ab23d0"}
{"status":"Download complete","progressDetail":{},"id":"c39877ab23d0"}
{"status":"Extracting","progressDetail":{"current":1179648
test_suite_pr (amd64, kvm) / Smoke tests (ext4, true): tests/eclient/testdata/metadata.txt#L17[stdout]
Docker app's state test
=== RUN TestAppStatus
apps: '[eclient]' state: 'RUNNING' secs: 1200
time: 2024-09-13T15:49:39.378762534Z out: appName eclient state changed to UNKNOWN
time: 2024-09-13T15:49:43.961104112Z out: appName eclient state changed to RESOLVING_TAG
time: 2024-09-13T15:49:48.96584475Z out: appName eclient state changed to DOWNLOAD_STARTED
time: 2024-09-13T15:49:53.969095065Z out: appName eclient state changed to DOWNLOAD_STARTED (0%)
time: 2024-09-13T15:49:57.971929899Z out: appName eclient state changed to DOWNLOAD_STARTED (100%)
time: 2024-09-13T15:50:02.9761247Z out: appName eclient state changed to DOWNLOAD_STARTED (0%)
time: 2024-09-13T15:50:09.982608403Z out: appName eclient state changed to DOWNLOAD_STARTED (2%)
time: 2024-09-13T15:50:14.987343607Z out: appName eclient state changed to DOWNLOAD_STARTED (9%)
time: 2024-09-13T15:50:16.989044461Z out: appName eclient state changed to DOWNLOAD_STARTED (19%)
time: 2024-09-13T15:50:19.992030773Z out: appName eclient state changed to DOWNLOAD_STARTED (33%)
time: 2024-09-13T15:50:21.994123877Z out: appName eclient state changed to DOWNLOAD_STARTED (56%)
time: 2024-09-13T15:50:21.994159243Z out: appName eclient state changed to DOWNLOAD_STARTED (73%)
time: 2024-09-13T15:50:22.995290843Z out: appName eclient state changed to DOWNLOAD_STARTED (78%)
time: 2024-09-13T15:50:24.997933765Z out: appName eclient state changed to DOWNLOAD_STARTED (82%)
time: 2024-09-13T15:50:24.997969543Z out: appName eclient state changed to DOWNLOAD_STARTED (92%)
time: 2024-09-13T15:50:30.002137451Z out: appName eclient state changed to DOWNLOAD_STARTED (97%)
time: 2024-09-13T15:50:33.005133046Z out: appName eclient state changed to LOADING
time: 2024-09-13T15:50:42.01379143Z out: appName eclient state changed to CREATING_VOLUME
time: 2024-09-13T15:51:14.035830619Z out: appName eclient state changed to INSTALLED
time: 2024-09-13T15:55:30.230661578Z out: appName eclient state changed to UNKNOWN
time: 2024-09-13T15:55:35.234631666Z out: appName eclient state changed to INITIAL
time: 2024-09-13T15:55:53.24673608Z out: appName eclient state changed to INSTALLED
time: 2024-09-13T15:55:57.249884422Z out: appName eclient state changed to INSTALLED: [description:"setting up OCI spec for domain 56d25a38-04b4-461b-83ee-34717eeb0d3f.1.1 failed unexpected end of JSON input" timestamp:{seconds:1726242956 nanos:259495873} severity:SEVERITY_ERROR]
app_test.go:178: ASSERTION FAILED (2024-09-13T16:09:39.379339292Z): expected apps [eclient] in RUNNING state
app_test.go:180: actual eclient: INSTALLED: [description:"setting up OCI spec for domain 56d25a38-04b4-461b-83ee-34717eeb0d3f.1.1 failed unexpected end of JSON input" timestamp:{seconds:1726242956 nanos:259495873} severity:SEVERITY_ERROR]
app_test.go:182: history of states for eclient:
app_test.go:184: state: no info from controller received in: 2024-09-13T15:49:39.375690839Z
app_test.go:184: state: UNKNOWN received in: 2024-09-13T15:49:39.378761091Z
app_test.go:184: state: RESOLVING_TAG received in: 2024-09-13T15:49:43.961099985Z
app_test.go:184: state: DOWNLOAD_STARTED received in: 2024-09-13T15:49:48.965840011Z
app_test.go:184: state: DOWNLOAD_STARTED (0%) received in: 2024-09-13T15:49:53.969091588Z
app_test.go:184: state: DOWNLOAD_STARTED (100%) received in: 2024-09-13T15:49:57.971927986Z
app_test.go:184: state: DOWNLOAD_STARTED (0%) received in: 2024-09-13T15:50:02.976121725Z
app_test.go:184: state: DOWNLOAD_STARTED (2%) received in: 2024-09-13T15:50:09.982606579Z
app_test.go:184: state: DOWNLOAD_STARTED (9%) received in: 2024-09-13T15:50:14.987338297Z
app_test.go:184: state: DOWNLOAD_STARTED (19%) received in: 2024-09-13T15:50:16.989042948Z
app_test.go:184: state: DOWNLOAD_STARTED (33%) received in: 2024-09-13T15:50:19.992028869Z
app_test.go:184: state: DOWNLOAD_STARTED (56%) received in: 2024-09-13T15:50:21.994121223Z
app_test.go:184: state: DOWNLOAD_STARTED (73%) received in: 2024-09-13T15:50:21.994157991Z
test_suite_pr (amd64, kvm) / User apps test suite: tests/app/testdata/app_replace_test.txt#L35[stdout] time="2024-09-13T16:13:06Z" level=error msg="cannot obtain info about volumes: error getting config library/nginx:latest: reading image \"index.docker.io/library/nginx:latest\": GET https://index.docker.io/v2/library/nginx/manifests/sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" time="2024-09-13T16:13:07Z" level=error msg="cannot obtain info about disks: GET https://index.docker.io/v2/library/nginx/manifests/latest: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" config changed, to see config run 'eden controller edge-node get-config' time="2024-09-13T16:13:08Z" level=info msg="deploy pod app2 with docker://nginx request sent" > exec -t 5m bash wait_for_app_error.sh app2 'no free IP addresses in DHCP range' [context deadline exceeded] FAIL: ../app/testdata/app_replace_test.txt:35: command failure |
|
https://github.com/lf-edge/eve/actions/runs/11103373032?pr=4284 test_suite_pr (amd64, kvm, generic) / Smoke tests (ext4, true): tests/eclient/testdata/ctrl_cert_change.txt#L61[stdout]
Network's state test
=== RUN TestNetworkStatus
nw_test.go:132: time: 2024-09-30T11:08:29.036269159Z out: networks: '[n1]' expected state: '-' secs: 600
test_suite_pr (amd64, kvm, generic) / Smoke tests (zfs, false): tests/eclient/testdata/ctrl_cert_change.txt#L42[stdout] Docker app's state test === RUN TestAppStatus apps: '[eclient1 eclient2]' state: 'RUNNING' secs: 1200 time: 2024-09-30T10:52:17.42679109Z out: appName eclient1 state changed to UNKNOWN time: 2024-09-30T10:52:17.4267964Z out: appName eclient2 state changed to UNKNOWN time: 2024-09-30T10:52:23.870147474Z out: appName eclient2 state changed to HALTING time: 2024-09-30T10:52:23.870162462Z out: appName eclient1 state changed to HALTING time: 2024-09-30T10:52:29.874538906Z out: appName eclient1 state changed to HALTED time: 2024-09-30T10:52:29.874553974Z out: appName eclient2 state changed to HALTED time: 2024-09-30T10:56:12.050109008Z out: appName eclient1 state changed to UNKNOWN time: 2024-09-30T10:56:12.050116372Z out: appName eclient2 state changed to UNKNOWN time: 2024-09-30T10:56:19.054566075Z out: appName eclient1 state changed to INITIAL time: 2024-09-30T10:56:19.054578428Z out: appName eclient2 state changed to INITIAL time: 2024-09-30T10:57:11.096219696Z out: appName eclient2 state changed to INSTALLED time: 2024-09-30T10:57:11.09624288Z out: appName eclient1 state changed to INSTALLED app_test.go:178: ASSERTION FAILED (2024-09-30T11:12:17.427794885Z): expected apps [eclient1 eclient2] in RUNNING state app_test.go:180: actual eclient1: INSTALLED app_test.go:182: history of states for eclient1: app_test.go:184: state: no info from controller received in: 2024-09-30T10:52:17.426786585Z app_test.go:184: state: UNKNOWN received in: 2024-09-30T10:52:17.426789307Z app_test.go:184: state: HALTING received in: 2024-09-30T10:52:23.87016114Z app_test.go:184: state: HALTED received in: 2024-09-30T10:52:29.874534798Z app_test.go:184: state: UNKNOWN received in: 2024-09-30T10:56:12.050105652Z app_test.go:184: state: INITIAL received in: 2024-09-30T10:56:19.054561436Z app_test.go:184: state: INSTALLED received in: 2024-09-30T10:57:11.096241517Z --- app eclient1 logs --- {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Found user 'avahi' (UID 86) and group 'avahi' (GID 86).","msgid":"1","timestamp":"2024-09-30T10:51:21.213866401Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Successfully dropped root privileges.","msgid":"2","timestamp":"2024-09-30T10:51:21.221708691Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"avahi-daemon 0.8 starting up.","msgid":"3","timestamp":"2024-09-30T10:51:21.221785960Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"WARNING: No NSS support for mDNS detected, consider installing nss-mdns!","msgid":"4","timestamp":"2024-09-30T10:51:21.223581799Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Successfully called chroot().","msgid":"5","timestamp":"2024-09-30T10:51:21.232938602Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Successfully dropped remaining capabilities.","msgid":"6","timestamp":"2024-09-30T10:51:21.233029271Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Loading service file /services/http.service.","msgid":"7","timestamp":"2024-09-30T10:51:21.239473077Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Loading service file /services/sftp-ssh.service.","msgid":"8","timestamp":"2024-09-30T10:51:21.249948537Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Loading service file /services/ssh.service.","msgid":"9","timestamp":"2024-09-30T10:51:21.260427867Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Joining mDNS multicast group on interface nbu1x1.1.IPv4 with address 10.11.12.2.","msgid":"10","timestamp":"2024-09-30T10:51:21.275112818Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"New relevant interface nbu1x1.1.IPv4 for mDNS.","msgid":"11","timestamp":"2024-09-30T10:51:21.277582510Z"} {"source":"018478a5-fc45-44ea-99a8-6bb895000e00.1.1","content":"Joining mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.","msgid":"12","timestamp":"2024-09-30T10:51:21.279391499Z"} {"source":"018478a5-fc45-44ea-99 |
This issue is created to collect and track information about flaky tests in our GitHub CI pipeline. Flaky tests are those that fail intermittently due to infrastructure or environment issues, rather than consistent failures in the tests or code. The goal is to distinguish real test failures from those caused by external factors and to identify any common patterns or problems that might be contributing to the flaky behavior.
The text was updated successfully, but these errors were encountered: