Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infrequent flake in golang_integration_test #29496

Closed
phlax opened this issue Sep 7, 2023 · 7 comments
Closed

Infrequent flake in golang_integration_test #29496

phlax opened this issue Sep 7, 2023 · 7 comments
Assignees
Labels
area/test flakes stale stalebot believes this issue/PR has not been touched recently

Comments

@phlax
Copy link
Member

phlax commented Sep 7, 2023

This doesnt seem to happen very often but its persisting

Ive mentioned it a few times on the golang slack channel but raising here as i think we need to look further

example fail:

[2023-09-07 17:22:55.282][120290][critical][golang] [contrib/golang/common/log/cgo.cc:27] log test
[2023-09-07 17:22:55.282][120290][error][golang] [contrib/golang/common/log/cgo.cc:24] log test false
[2023-09-07 17:22:55.282][120290][critical][golang] [contrib/golang/common/log/cgo.cc:27] log test false
[2023-09-07 17:23:05.285][119877][critical][assert] [test/integration/http_integration.cc:550] assert failure: 0. Details: Timed out waiting for new connection.
[2023-09-07 17:23:05.285][119877][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3e90001d445
[2023-09-07 17:23:05.285][119877][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2023-09-07 17:23:05.285][119877][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.28.0-dev/test/RELEASE/BoringSSL
[2023-09-07 17:23:05.286][119877][critical][backtrace] [./source/server/backtrace.h:96] #0: __kernel_rt_sigreturn [0xffffb63088dc]
[2023-09-07 17:23:05.286][119877][critical][backtrace] [./source/server/backtrace.h:96] #1: abort [0xffffb607daac]
[2023-09-07 17:23:05.313][119877][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::HttpIntegrationTest::waitForNextUpstreamConnection() [0x2ad7d6c]
[2023-09-07 17:23:05.361][119877][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::HttpIntegrationTest::waitForNextUpstreamRequest() [0x2ad74bc]
[2023-09-07 17:23:05.392][119877][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::HttpIntegrationTest::waitForNextUpstreamRequest() [0x2ad7fd4]
[2023-09-07 17:23:05.413][119877][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::GolangIntegrationTest::testDynamicMetadata() [0x13bdbfc]
[2023-09-07 17:23:05.434][119877][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::GolangIntegrationTest_DynamicMetadata_Async_Sleep_Test::TestBody() [0x13be640]
[2023-09-07 17:23:05.454][119877][critical][backtrace] [./source/server/backtrace.h:96] #7: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x3c32420]
[2023-09-07 17:23:05.475][119877][critical][backtrace] [./source/server/backtrace.h:96] #8: testing::Test::Run() [0x3c322c0]
[2023-09-07 17:23:05.496][119877][critical][backtrace] [./source/server/backtrace.h:96] #9: testing::TestInfo::Run() [0x3c33410]
[2023-09-07 17:23:05.516][119877][critical][backtrace] [./source/server/backtrace.h:96] #10: testing::TestSuite::Run() [0x3c3417c]
[2023-09-07 17:23:05.537][119877][critical][backtrace] [./source/server/backtrace.h:96] #11: testing::internal::UnitTestImpl::RunAllTests() [0x3c41684]
[2023-09-07 17:23:05.557][119877][critical][backtrace] [./source/server/backtrace.h:96] #12: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x3c41138]
[2023-09-07 17:23:05.578][119877][critical][backtrace] [./source/server/backtrace.h:96] #13: testing::UnitTest::Run() [0x3c40f94]
[2023-09-07 17:23:05.598][119877][critical][backtrace] [./source/server/backtrace.h:96] #14: Envoy::TestRunner::RunTests() [0x35c054c]
[2023-09-07 17:23:05.619][119877][critical][backtrace] [./source/server/backtrace.h:96] #15: main [0x35bf0b4]
[2023-09-07 17:23:05.619][119877][critical][backtrace] [./source/server/backtrace.h:96] #16: __libc_start_main [0xffffb607de10]
================================================================================
FAIL: //contrib/golang/filters/http/test:golang_integration_test (see /build/bazel_root/base/execroot/envoy/bazel-out/aarch64-opt/testlogs/contrib/golang/filters/http/test/golang_integration_test/test.log)
INFO: From Testing //contrib/golang/filters/http/test:golang_integration_test:
INFO: Elapsed time: 2725.781s, Critical Path: 553.54s
INFO: 34540 processes: 15507 remote cache hit, 12759 internal, 1 local, 6273 processwrapper-sandbox.
//contrib/golang/filters/http/test:golang_integration_test               FAILED in 21.9s

https://dev.azure.com/cncf/envoy/_build/results?buildId=148510&view=logs&j=767be981-567e-57d8-68c3-2140ede0a0bd&t=8a34f9ed-4946-5fcf-7cd6-99c93070884d&l=1473

@phlax
Copy link
Member Author

phlax commented Sep 7, 2023

cc @doujiang24

@doujiang24
Copy link
Member

Okay, I'll look further and keep an eye on it, thanks~

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale stalebot believes this issue/PR has not been touched recently label Oct 28, 2023
Copy link

github-actions bot commented Nov 4, 2023

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 4, 2023
@ravenblackx
Copy link
Contributor

Still happening (link)

@ravenblackx ravenblackx reopened this Jan 22, 2024
@github-actions github-actions bot removed the stale stalebot believes this issue/PR has not been touched recently label Jan 23, 2024
jmarantz pushed a commit that referenced this issue Jan 30, 2024
fix #31654 and #29496

Thanks @spacewander for founding this race:
envoyGoRequestSemaDec may be invoked eariler than r.sema.Wait, then, there is no one to resume the r.sema.Wait.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
Fixes: #31654, #29496


Signed-off-by: doujiang24 <doujiang24@gmail.com>
Copy link

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale stalebot believes this issue/PR has not been touched recently label Feb 22, 2024
Copy link

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 29, 2024
spacewander pushed a commit to mosn/envoy that referenced this issue Jun 27, 2024
…roxy#32081)

fix envoyproxy#31654 and envoyproxy#29496

Thanks @spacewander for founding this race:
envoyGoRequestSemaDec may be invoked eariler than r.sema.Wait, then, there is no one to resume the r.sema.Wait.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
Fixes: envoyproxy#31654, envoyproxy#29496


Signed-off-by: doujiang24 <doujiang24@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/test flakes stale stalebot believes this issue/PR has not been touched recently
Projects
None yet
Development

No branches or pull requests

3 participants