Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaky Test] tests/python/contrib/test_ethosu/test_replace_depthwise_conv2d.py::test_depthwise_conv2d_single[trial3] #12841

Closed
driazati opened this issue Sep 19, 2022 · 2 comments
Closed

[Flaky Test] tests/python/contrib/test_ethosu/test_replace_depthwise_conv2d.py::test_depthwise_conv2d_single[trial3] #12841

driazati opened this issue Sep 19, 2022 · 2 comments
Labels
byoc:ethosu test: flaky

Comments

@driazati
Copy link
Member

These tests were found to be flaky (intermittently failing on main or failed in a PR with unrelated changes). See the docs for details.

Tests(s)

  • tests/python/contrib/test_ethosu/test_replace_depthwise_conv2d.py::test_depthwise_conv2d_single[trial3]
[2022-09-19T18:44:28.351Z] tests/python/contrib/test_ethosu/test_replace_depthwise_conv2d.py::test_depthwise_conv2d_single[trial3] Fatal Python error: Segmentation fault
[2022-09-19T18:44:28.351Z] 
[2022-09-19T18:44:28.351Z] Thread 0x00007fc47f440700 (most recent call first):
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 400 in read
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 432 in from_io
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
[2022-09-19T18:44:28.351Z] 
[2022-09-19T18:44:28.351Z] Current thread 0x00007fc48157a740 (most recent call first):
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/ethosu/vela/weight_compressor.py", line 174 in encode_weights
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/ethosu/vela/api.py", line 431 in npu_encode_weights
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/vela_api.py", line 232 in compress_weights
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/vela_api.py", line 168 in encode_weights
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/passes.py", line 353 in _encode_weights
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/passes.py", line 383 in _encode_weights_or_bias
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/passes.py", line 444 in _visit
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 81 in cfun
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/tir/stmt_functor.py", line 59 in post_order_visit
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/passes.py", line 466 in collect_encoding_definitions
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/passes.py", line 567 in _ftransform
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 81 in cfun
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/ir/transform.py", line 161 in __call__
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/passes.py", line 655 in _encode_constants
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/compiler.py", line 91 in lower_ethosu
[2022-09-19T18:44:28.351Z]   File "/workspace/python/tvm/relay/backend/contrib/ethosu/tir/compiler.py", line 251 in _lower_to_tir
[2022-09-19T18:44:28.351Z]   File "/workspace/tests/python/contrib/test_ethosu/test_replace_depthwise_conv2d.py", line 111 in test_depthwise_conv2d_single
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in __call__
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/python.py", line 1761 in runtest
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/runner.py", line 166 in pytest_runtest_call
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in __call__
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/runner.py", line 259 in <lambda>
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/runner.py", line 338 in from_call
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/runner.py", line 219 in call_and_report
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/runner.py", line 130 in runtestprotocol
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pytest_rerunfailures.py", line 497 in pytest_runtest_protocol
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in __call__
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/xdist/remote.py", line 110 in run_one_test
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/xdist/remote.py", line 91 in pytest_runtestloop
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in __call__
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/main.py", line 322 in _main
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/main.py", line 268 in wrap_session
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in __call__
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/xdist/remote.py", line 291 in <module>
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1084 in executetask
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1060 in serve
[2022-09-19T18:44:28.351Z]   File "/opt/tvm-venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1554 in serve
[2022-09-19T18:44:28.351Z]   File "<string>", line 8 in <module>
[2022-09-19T18:44:28.351Z]   File "<string>", line 1 in <module>

Jenkins Links
@driazati driazati mentioned this issue Sep 30, 2022
Merged
driazati added a commit to driazati/tvm that referenced this issue Sep 30, 2022
@driazati
[ci] Disable flaky ethosu + roofline tests
6f8a420 
These are all segfaulting in main (see apache#12955, apache#12933, and apache#12841) so
they need to be skipped until a fix is merged.
@driazati driazati mentioned this issue Sep 30, 2022
Closed
driazati added a commit to driazati/tvm that referenced this issue Sep 30, 2022
@driazati
[ci] Disable flaky ethosu + roofline tests
411109a 
These are all segfaulting in main (see apache#12955, apache#12933, and apache#12841) so
they need to be skipped until a fix is merged.
driazati added a commit to driazati/tvm that referenced this issue Sep 30, 2022
@driazati
[ci] Disable flaky ethosu + roofline tests
5f6ef00 
These are all segfaulting in main (see apache#12955, apache#12933, and apache#12841) so
they need to be skipped until a fix is merged.
driazati added a commit to driazati/tvm that referenced this issue Sep 30, 2022
@driazati
[ci] Disable flaky ethosu + roofline tests
9748664 
These are all segfaulting in main (see apache#12955, apache#12933, and apache#12841) so
they need to be skipped until a fix is merged.
driazati added a commit to driazati/tvm that referenced this issue Sep 30, 2022
@driazati
[ci] Disable flaky ethosu + roofline tests
b81f79e 
These are all segfaulting in main (see apache#12955, apache#12933, and apache#12841) so
they need to be skipped until a fix is merged.
driazati added a commit to driazati/tvm that referenced this issue Oct 3, 2022
@driazati
[ci] Disable flaky ethosu + roofline tests
c1965ad 
These are all segfaulting in main (see apache#12955, apache#12933, and apache#12841) so
they need to be skipped until a fix is merged.
junrushao pushed a commit that referenced this issue Oct 6, 2022
@driazati
[ci] Disable flaky ethosu + roofline tests (#12956)
3d5e440 
These are all segfaulting in main (see #12955, #12933, and #12841) so
they need to be skipped until a fix is merged.

Co-authored-by: driazati <driazati@users.noreply.github.com>
@ekalda
Copy link
Contributor

ekalda commented Oct 12, 2022

It looks like this is the caused by the same issue of segfaulting when calling Vela's weight encoder for which there are few issues raised and a fix in progress - #12634 (comment)

@areusch areusch added the needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it label Oct 19, 2022
@janetsc janetsc added byoc:ethosu and removed needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it labels Oct 19, 2022
xinetzone pushed a commit to daobook/tvm that referenced this issue Nov 25, 2022
@driazati @xinetzone
[ci] Disable flaky ethosu + roofline tests (apache#12956)
99d00ef 
These are all segfaulting in main (see apache#12955, apache#12933, and apache#12841) so
they need to be skipped until a fix is merged.

Co-authored-by: driazati <driazati@users.noreply.github.com>
@lhutton1
Copy link
Contributor

lhutton1 commented Nov 3, 2023

closing, same issue as #12634

@lhutton1 lhutton1 closed this as completed Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
byoc:ethosu test: flaky
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@areusch @driazati @ekalda @janetsc @lhutton1