CI monitoring rotation schedule

[driazati]

See the CI Monitoring Runbook for context.

This schedule tracks the CI monitoring rotation. If you would like to join, please comment on this issue or, if you are a committer, edit this issue directly.

Week (MM/DD)	On-call
8/1 - 8/5	@driazati
8/8 - 8/12	@gigiblender
8/15 - 8/19	@shingjan
8/22 - 8/26	@cconvey
8/29 - 9/2	@sunggg
9/5 - 9/9	@driazati
9/12 - 9/16	@nverke
9/19 - 9/23	---
9/26 - 9/30	---
10/3 - 10/7	---

[driazati]

This isn't part of the regular rotation but I thought I'd post a summary of my week so everyone has some more visibility into the process. Last week was lots of infra problems which hopefully are fixed now that we've increased capacity limits and fixed our cleanup logic.

• jenkins on c253053 - reverted in [skip ci] Revert "[skip ci][ci][docker] Prune all non-relevant images (#11491)" #11496
• jenkins on 01ee1bc - failed due to a Jenkins reboot
• jenkins on 6f3c8bd - failed due to a Jenkins reboot
• jenkins on 7766ab2 - failed due to a Jenkins reboot
• jenkins on 2a2d910 - failed due to a Jenkins reboot
• jenkins on 6895087 - network problems, filed Use git annex to log large files in CI #11492
• jenkins on cfcca59 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on 2f21698 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on b535e46 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on 52df2e8 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on a9ece3d - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on 8135860 - timed out due to queuing
• jenkins on d519b03 - timed out due to queuing
• jenkins on 92cc5b0 - timed out due to queuing
• jenkins on bbdb656 - network problems and flaky test, filed [ci][network error] tests/python/frontend/tflite/test_forward.py::test_forward_tflite_float16, tests/python/frontend/tflite/test_forward.py::test_forward_qnn_inception_v1_net #11459 and [Flaky Test] tests/python/unittest/test_arith_solve_linear_inequality.py::test_solution_consistency #11458
• jenkins on 6c6dfbc - fixed by [ci] Increase timeout for git submodule update #11457
• jenkins on c247295 - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 014208e - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 3f53e7a - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 7e83c4a - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 7ba8a61 - failed due to a
• jenkins on b141cac - filed [Flaky Test] tests/python/unittest/test_auto_scheduler_evolutionary_search.py::test_mutate_parallel #11440
• gha on 7bab8f7 - filed [ci] MacOS GitHub Actions occasionally dies #11416

[areusch]

Summary from my week: things were fairly quiet, mainly some flaky tests.

• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3414/pipeline) on c1b22ee - flaky test, filed [Flaky Test] tests/python/contrib/test_ethosu/test_codegen.py::test_mean[ifm_shape0-axis0-True-False-ethos-u65-256] #11527
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3415/pipeline) on ac5d781 - continued flaky paddlepaddle test, updated [Flaky] Possible flaky tests from Paddle #9976. better diagnostic submitted today.
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3411/pipeline) on bc14f26 - network flake, filed [CI Problem] from_oneflow can flake due to network dependency #11514
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3444/tests) on 2ae2088 - assert_allclose 2/10000 mismatched (0.02%), filed [Flaky Test] tests/python/frontend/pytorch/test_forward.py::test_argsort #11568

[hpanda-naut]

@areusch @driazati I can take next week

[hpanda-naut]

Summary from 6/13/22 - 6/19/22: mostly infra and flaky test issues

• jenkins on 2df4524 [ci] docker pull network errors #11710
• jenkins on 81cc086 [ci] docker pull failed #11713
• gha on a82d2f5 [ci] andriod_rpc build failed #11738
• jenkins on https://github.com/dos-lab/as-tvm/commit/583696fceb8abd4c3c2f4abb93d0dbce2e79f60f [ci] pytest_wrapper needs guards for invalid JUnits #11749
• jenkins [Flaky Test] tests/python/contrib/test_ethosu/test_codegen.py::test_mean[ifm_shape0-axis0-True-False-ethos-u55-256] #11763
• jenkins on b4a77ac [ci] test wrapper scripts failure #11764
• jenkins on 65d45af [bug][docs] error in "code" directive: maximum 1 argument(s) allowed, 2 supplied. #11804
• gha on 0fdc0ea [ci] Execution failed for task ':app:checkDebugAarMetadata' on android build #11805

[gigiblender]

Happy to sign up for 6/27-7/1 if someone could edit the issue for me :)

[nverke]

Week of 6/20 - 6/27:

• Details on bc75487 [Flaky Test] tests/python/contrib/test_ethosu/test_codegen.py::test_mean[ifm_shape2-axis2-True-False-ethos-u65-256] #11830
• Details on b63801c [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on a363a04 [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on 7b0f791 [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on a5366a7 [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on 32d16eb [CI Problem] Failure while downloading files from S3. #11917
• Details on 98fb955 [CI Problem] No space left on device #11840
• Details on 6ed3ab3 [CI Problem] No space left on device #11840
• Details on 1e0e954 (Fixed not sure what the issue was)
• Details on 51b0d8c [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #11580
• Details on 410e836 [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #11580
• Details on 12e8744 [CI Problem] Docs failed to build with code 143 #11918
• Details on 5e81067 [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #11580
• Details on 4cb18b4 [CI Problem] /workspace/gallery/how_to/compile_models/from_oneflow.py failed due to url issue.  #11919
• Details on aa66e9f [CI Problem] Docs failed to build with code 143 #11918
• Details on 600a201 [CI Problem] No space left on device #11840
• Details on 98bf40f [CI Problem] Jenkins link 404  #11920

[sunggg]

I can help on 7/11-7/15 :)

[gigiblender]

27/06/2022
jenkins on 1115fd9 aborted due to CI disk space issue

28/06/2022
jenkins on 6c8a353 #11580
jenkins on 688b082 #11580

29/06/2022
jenkins on a17bfc0 #11964

30/06/2022
jenkins on 41c94b2 #11967
jenkins on 898946f #11567
jenkins on 558ba99 #12004 and flaky due to network
jenkins on e7851ed #11967
jenkins on 80a0c6c #11967
jenkins on 915c23b #11967
jenkins on c0f4bf7 #11967

01/07/2022
jenkins on ec39199 #11580
jenkins on 395e91f #11580
jenkins on 9e14509 #11568
jenkins on 50cd4d6 #11568

[areusch]

• [jenkins](https://ci.tlcpack.ai/blue/organizations/[jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3755/pipeline)/tvm/detail/main/3753/pipeline/) on cfe8318 - failed to docker pull from DockerHub
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3755/pipeline) on 9f4bf38 - failed to docker pull from DockerHub
• jenkins on 99d42b2 - test_qlinearadd tolerance, filed [Flaky Test] tests/python/frontend/onnx/test_forward.py::test_qlinearadd[cuda] #12062
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3767/pipeline) on f769f4e - Skipped has no len, added to [ci] pytest_wrapper needs guards for invalid JUnits #11749
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3769/pipeline) on a81e69a - http flake

[sunggg]

• jenkins on ef5c3ed - failure on git push to https://github.com/apache/tvm-site.git
• jenkins on 3992d24 - numerical error at test_myloat() in test_custom_datatypes.py, filed [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #12092
• jenkins on b9fa576 - Error response from daemon: Get "https://registry-1.docker.io/v2/tlcpack/ci-i386/manifests/sha256:096c51de2a27fd3e497c17433937a4cd4357fe9623f06f6b2a47e094cf515b14": EOF

[yongwww]

Summary for 7/25/22 - 7/29/22:

jenkins on eada707 - failure on android-rpc build
jenkins on 88bbb40 - failure on git push to https://github.com/apache/tvm-site.git
jenkins on 03aed78 - failure on ios-rpc
jenkins on 195e60b - failure on duplicate global packedfunc
jenkins on 9a4d80c - filed [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #12238
jenkins on aeda760 - filed #12238
jenkins on 578ef03 - filed #12238

[driazati]

Summary for 2022-07-30 to 2022-08-06

Run	Commit	Mitigation
tvm-ci/branch	1f97f1f from #11809	Timeout fixed by #12334
tvm-ci/branch	485bfaf from #12306	Fixed by #12325
tvm-ci/branch	4158738 from #12301	Flaky test, filed #12311 and opened #12312 to improve reporting
tvm-ci/branch	2bfd52f from #12278	Fixed by #12282
tvm-ci/branch	12502cc from #12251	Fixed by #12268
tvm-ci/branch	a231a1d from #12245	Added retries to fix: #12306
tvm-ci/branch	fb87c21 from #12234	Fixed by #12306
tvm-ci/branch	dff5c97 from #11037	Fixed by #12306
tvm-ci/branch	db4380c from #12230	Fixed by #12268

_{Generated by https://gist.github.com/driazati/80cd48e86c6548cd90a6b39be010b921}

[gigiblender]

Summary for 2022-08-07 - 2022-08-14:
jenkins on fc411dc - Failure due to timeout in Cortex-M shards. Fixed in #12334.
jenkins on 9b86009 - Failure due to timeout in Cortex-M shards. Fixed in #12334.
jenkins on 8e133b1 - Sent interrupt signal to the CI.
jenkins on 7f800e4 - Fixed by #12341.
CI MacOS on 52d6b59 - Filed #12449.
jenkins on 2210206 - Failure due to linting the Jenkinsfile. Fixed by #12360.
jenkins on 5d72bc1 - Failure due to linting the Jenkinsfile. Fixed by #12360.
jenkins on 7f10015 - Failure due to linting the Jenkinsfile. Fixed by #12360.
jenkins on 52152e0 - Failure due to linting the Jenkinsfile. Fixed by #12387.
jenkins on 48354de - Failure due to linting the Jenkinsfile. Fixed by #12387.
jenkins on 5deb95a - Deploy docs failed to git push in the CI.
jenkins on c3c7c4c - Failed due to flaky test. #12451.
jenkins on 3eb6734 - Failed due to flaky test. #12451.
jenkins on 1737308 - Linting failed with exit code 4.

[shingjan]

Summary for 2022-08-15 to 2022-08-22

Run	Commit	Mitigation
tvm-ci/branch	d805ae3 from #12425	Internet errors - flaky test reported in #12465
tvm-ci/branch	1ba17fe from #12401	Timeout errors - flaky test reported in #12464
tvm-ci/branch	bd56231 from #12443	Timeout errors - flaky test reported in #12464
tvm-ci/branch	#12478	flaky test reported in #12511
tvm-ci/branch	#12441	flaky test reported in #12511
tvm-ci/branch	#12483	flaky test reported in #12511
tvm-ci/branch	#12513	flaky test reported in #12511
tvm-ci/branch	#12532	flaky test reported in #12511
tvm-ci/branch	#12508	flaky test reported in #12511
tvm-ci/branch	#12551	flaky test reported in #12511
tvm-ci/branch	#12539	doc build failed

_{Generated by https://gist.github.com/driazati/80cd48e86c6548cd90a6b39be010b921}

[cconvey]

Summary for 2022-08-22 to 2022-08-29

Run	Commit	Mitigation
tvm-ci/branch	1afd059 from #12340	Checks API failure (#12602)
tvm-ci/branch	90b2f0d from #12557	maven problem (#12601)
CI / Android	8174d08	android_rpc build failure (#12599)
tvm-ci/branch	13ebbfb from #12562	Deploy docs failed to git push in the CI (#12600)
tvm-ci/branch	52779f1 from #12353	ethos-u failures (#12511)
tvm-ci/branch	3983a47 from #12543	ethos-u failures (#12511)
tvm-ci/branch	d26bf80 from #12541	ethos-u failures (#12511)
tvm-ci/branch	5344128 from PR #12623	internal pytest failure during ethos-u testing (#12634)

_{Generated by https://gist.github.com/driazati/80cd48e86c6548cd90a6b39be010b921}

[sunggg]

Summary for 8/29 - 9/2

jenkins on 0de2219 - Unable to find image 'tlcpack/ci-lint:20220810-060142-fae79bbc3' locally
jenkins on 0de2219 - Segmentation fault (core dumped)
jenkins on 74988d3 - Failed unit tests
jenkins on 58ee935 - Failed unit tests
jenkins on a399e6c - Failed unit tests
jenkins on aa6c7123d0a2cdd93256c6a4576ff029008fd375- segfault in tests/scripts/setup-pytest-env.sh
jenkins on 50dad0d - failed to push due to merge conflcit
jenkins on eecb7fd - segfault in tests/scripts/setup-pytest-env.sh
jenkins on b2d6600 - ERROR tests/python/frontend/darknet/test_forward.py - urllib.error.HTTPError
jenkins on bb56f2a - http error 502 on ``tests/python/frontend/tflite/test_forward.py`
jenkins on 0549a08 - urllib.error.HTTPError: HTTP Error 503: Service Unavailable