Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

Merged
merged 7 commits into from
May 12, 2023
Merged

Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

merged 7 commits into from
May 12, 2023

Conversation

yanjunxiang-google
Copy link
Contributor

@yanjunxiang-google yanjunxiang-google commented May 8, 2023 •
edited
Loading

Properly handle dataplane exception when received response timeout Duration is too large.

Also adding PGVs for ext_proc filter config.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

@yanjunxiang-google
Adding ext_proc filter config and response timeout Duration PGVs to a…
adb6e5a 
…void ext_proc filter fuzzer crash due to duration config out-of-bounds.

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@yanjunxiang-google yanjunxiang-google requested a review from snowp as a code owner May 8, 2023 19:21
@repokitteh-read-only
Copy link

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @mattklein123
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #27260 was opened by yanjunxiang-google.

see: more, trace.

@yanjunxiang-google
Copy link
Contributor Author

The crash traceback is:
./bazel-bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz crash-caae576f1c5a5c4bd6f831dd7d159b2504f476b0
INFO: found LLVMFuzzerCustomMutator (0x298ec70). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3719293196
INFO: Loaded 1 modules (1566644 inline 8-bit counters): 1566644 [0xae23d00, 0xafa24b4),
INFO: Loaded 1 PC tables (1566644 PCs): 1566644 [0xafa24b8,0xc789ff8),
./bazel-bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz: Running 1 inputs 1 time(s) each.
Running: crash-caae576f1c5a5c4bd6f831dd7d159b2504f476b0
terminate called after throwing an instance of 'Envoy::DurationUtil::OutOfRangeException'
what(): Duration out-of-range: seconds: 137438953472
nanos: 655360

==1818001== ERROR: libFuzzer: deadly signal
error: failed to decompress '.debug_aranges', zlib is not available
error: failed to decompress '.debug_info', zlib is not available
error: failed to decompress '.debug_abbrev', zlib is not available
error: failed to decompress '.debug_line', zlib is not available
error: failed to decompress '.debug_str', zlib is not available
error: failed to decompress '.debug_line_str', zlib is not available
error: failed to decompress '.debug_loclists', zlib is not available
error: failed to decompress '.debug_rnglists', zlib is not available
#0 0x2953121 in __sanitizer_print_stack_trace /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
#1 0x2870528 in fuzzer::PrintStackTrace() /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5
#2 0x28551e3 in fuzzer::Fuzzer::CrashCallback() /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3
#3 0x7f2193c1bf8f (/lib/x86_64-linux-gnu/libc.so.6+0x3bf8f) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#4 0x7f2193c6accb (/lib/x86_64-linux-gnu/libc.so.6+0x8accb) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#5 0x7f2193c1bef1 in raise (/lib/x86_64-linux-gnu/libc.so.6+0x3bef1) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#6 0x7f2193c06471 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x26471) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#7 0x7f219409d918 (/lib/x86_64-linux-gnu/libstdc++.so.6+0x9d918) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#8 0x7f21940a8e19 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa8e19) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#9 0x7f21940a8e84 in std::terminate() (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa8e84) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#10 0x7f21940a90d7 in __cxa_throw (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa90d7) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#11 0x69b6048 in Envoy::(anonymous namespace)::validateDuration(google::protobuf::Duration const&, long) /proc/self/cwd/source/common/protobuf/utility.cc:654:5
#12 0x69a82f2 in Envoy::DurationUtil::durationToMilliseconds(google::protobuf::Duration const&) /proc/self/cwd/source/common/protobuf/utility.cc:669:3
#13 0x2b6b64b in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__1::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__1::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse >&&) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:562:18
#14 0x2a8d7ed in std::__1::__function::__func<Envoy::Extensions::HttpFilters::ExtProc::UnitTestFuzz::TestOneProtoInput(envoy::extensions::filters::http::ext_proc::unit_test_fuzz::ExtProcUnitTestCase const&)::$_0::operator()(Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorCallbacks&, envoy::config::core::v3::GrpcService const&, Envoy::StreamInfo::StreamInfo const&) const::'lambda'(envoy::service::ext_proc::v3::ProcessingRequest&&, bool), std::__1::allocator<Envoy::Extensions::HttpFilters::ExtProc::UnitTestFuzz::TestOneProtoInput(envoy::extensions::filters::http::ext_proc::unit_test_fuzz::ExtProcUnitTestCase const&)::$_0::operator()(Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorCallbacks&, envoy::config::core::v3::GrpcService const&, Envoy::StreamInfo::StreamInfo const&) const::'lambda'(envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>, void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::operator()(envoy::service::ext_proc::v3::ProcessingRequest&&, bool&&) /proc/self/cwd/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc:81:31
#15 0x2a7af17 in std::__1::__function::__value_func<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::operator()(envoy::service::ext_proc::v3::ProcessingRequest&&, bool&&) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
#16 0x2a7adeb in std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::operator()(envoy::service::ext_proc::v3::ProcessingRequest&&, bool) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
#17 0x2a7acb2 in decltype(std::forward<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&>(fp)(std::get<0ul>(std::forward<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(fp0)), std::get<1ul>(std::forward<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(fp0)))) testing::internal::ApplyImpl<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>, 0ul, 1ul>(std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>&&, testing::internal::IndexSequence<0ul, 1ul>) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/internal/gmock-internal-utils.h:420:10
#18 0x2a7abf0 in decltype(ApplyImpl(std::forward<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&>(fp), std::forward<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(fp0), (testing::internal::MakeIndexSequence<std::tuple_size<std::__1::remove_reference<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >::type>::value>)())) testing::internal::Apply<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>&&) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/internal/gmock-internal-utils.h:429:10
#19 0x2a7a30d in testing::Action<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::Perform(std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>) const /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:497:12
#20 0x2a7b013 in testing::internal::ActionResultHolder* testing::internal::ActionResultHolder::PerformAction<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>(testing::Action<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, testing::internal::Function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::ArgumentTuple&&) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1441:12
#21 0x2a78c32 in testing::internal::FunctionMocker<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::UntypedPerformAction(void const*, void*) const /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1556:12
#22 0x8da6fa0 in testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*) /proc/self/cwd/external/com_google_googletest/googlemock/src/gmock-spec-builders.cc:452:24
#23 0x2a84c05 in testing::internal::FunctionMocker<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::Invoke(envoy::service::ext_proc::v3::ProcessingRequest&&, bool) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1593:15
#24 0x2a782ec in Envoy::Extensions::HttpFilters::ExtProc::UnitTestFuzz::MockStream::send(envoy::service::ext_proc::v3::ProcessingRequest&&, bool) /proc/self/cwd/./test/extensions/filters/http/ext_proc/unit_test_fuzz/mocks.h:20:3
#25 0x2b5a2c5 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onHeaders(Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState&, Envoy::Http::RequestOrResponseHeaderMap&, bool) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:176:12
#26 0x2b5b8d8 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:194:23
#27 0x2ad1855 in Envoy::Http::FilterHeadersStatus Envoy::Extensions::HttpFilters::HttpFilterFuzzer::sendHeadersEnvoy::Http::StreamDecoderFilter(Envoy::Http::StreamDecoderFilter*, test::fuzz::HttpData const&, bool) /proc/self/cwd/./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:145:46
#28 0x2995b55 in void Envoy::Extensions::HttpFilters::HttpFilterFuzzer::runDataEnvoy::Http::StreamDecoderFilter(Envoy::Http::StreamDecoderFilter*, test::fuzz::HttpData const&) /proc/self/cwd/./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:102:31
#29 0x298ffeb in LLVMFuzzerTestOneInput /proc/self/cwd/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc:88:10
#30 0x2856783 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
#31 0x2841212 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
#32 0x2846abc in fuzzer::FuzzerDriver(int*, char***, int ()(unsigned char const, unsigned long)) /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
#33 0x2870ce2 in main /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
#34 0x7f2193c07189 (/lib/x86_64-linux-gnu/libc.so.6+0x27189) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#35 0x7f2193c07244 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x27244) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#36 0x28373ad in _start (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/51ff81aa23c8ee714a5106cc912b2104/execroot/envoy/bazel-out/k8-dbg/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x28373ad)

NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal

@yanjunxiang-google
Copy link
Contributor Author

/assign @yanavlasov @adisuissa @htuch

@yanjunxiang-google
Adding exception handling for response with large override_message_t…
2554501 
…imeout

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
htuch
htuch previously approved these changes May 9, 2023
View reviewed changes
@yanjunxiang-google yanjunxiang-google changed the title Adding ext_proc filter config and response timeout Duration PGVs Envoy ext_proc filter throw exception when received response timeout Duration is too large May 9, 2023
adisuissa
adisuissa reviewed May 9, 2023
View reviewed changes
Copy link
Contributor

@adisuissa adisuissa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
I think the 1hr limit should be ok for most use-cases.
Left a small comment on the handling of the override as part of the response case.

source/extensions/filters/http/ext_proc/ext_proc.cc Outdated Show resolved Hide resolved
@mattklein123 mattklein123 removed their assignment May 9, 2023
yanavlasov
yanavlasov reviewed May 9, 2023
View reviewed changes
Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait-any

yanavlasov
yanavlasov requested changes May 9, 2023
View reviewed changes
Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait

source/extensions/filters/http/ext_proc/ext_proc.cc Outdated Show resolved Hide resolved
@yanjunxiang-google
addressing comments
f796853 
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
yanavlasov
yanavlasov requested changes May 10, 2023
View reviewed changes
Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait

source/common/protobuf/utility.cc Outdated Show resolved Hide resolved
@yanjunxiang-google
adding unit test for the newly added NoThrow utility function
b3187e1 
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@yanjunxiang-google
addressing comments
217b185 
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@yanjunxiang-google
addressing comments
9ccba29 
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
adisuissa
adisuissa reviewed May 11, 2023
View reviewed changes
Copy link
Contributor

@adisuissa adisuissa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! A few minor comments, but overall looks good.

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto Show resolved Hide resolved
source/common/protobuf/utility.cc Outdated Show resolved Hide resolved
source/common/protobuf/utility.cc Outdated Show resolved Hide resolved
source/common/protobuf/utility.cc Outdated Show resolved Hide resolved
source/common/protobuf/utility.cc Outdated Show resolved Hide resolved
source/extensions/filters/http/ext_proc/ext_proc.cc Outdated Show resolved Hide resolved
@yanjunxiang-google
addressing comments
1b25122 
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@yanavlasov yanavlasov enabled auto-merge (squash) May 12, 2023 15:04
@yanavlasov yanavlasov merged commit b14b3e8 into envoyproxy:main May 12, 2023
adisuissa
adisuissa reviewed May 12, 2023
View reviewed changes
Copy link
Contributor

@adisuissa adisuissa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
Let's revert the descriptions updates as they don't add anything, and we can approve this.

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto Show resolved Hide resolved
api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto Show resolved Hide resolved
wbpcode pushed a commit to wbpcode/envoy that referenced this pull request May 16, 2023
@yanjunxiang-google
Envoy ext_proc filter throw exception when received response timeout …
5790c93 
…Duration is too large (envoyproxy#27260)

* Adding ext_proc filter config and response timeout Duration PGVs to avoid ext_proc filter fuzzer crash due to duration config out-of-bounds.

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
reskin89 pushed a commit to reskin89/envoy that referenced this pull request Jul 11, 2023
@yanjunxiang-google @reskin89
Envoy ext_proc filter throw exception when received response timeout …
e749b66 
…Duration is too large (envoyproxy#27260)

* Adding ext_proc filter config and response timeout Duration PGVs to avoid ext_proc filter fuzzer crash due to duration config out-of-bounds.

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
Signed-off-by: Ryan Eskin <ryan.eskin89@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adisuissa adisuissa adisuissa left review comments

@yanavlasov yanavlasov yanavlasov approved these changes

@htuch htuch htuch left review comments

@snowp snowp Awaiting requested review from snowp

Assignees

@adisuissa adisuissa

@yanavlasov yanavlasov

@htuch htuch

Labels
api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yanjunxiang-google @adisuissa @yanavlasov @htuch @mattklein123