Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tcp_metadata_exchange proxy test fails on big-endian platforms #294

Closed
knm3000 opened this issue Jul 25, 2022 · 0 comments · Fixed by #282
Closed

tcp_metadata_exchange proxy test fails on big-endian platforms #294

knm3000 opened this issue Jul 25, 2022 · 0 comments · Fixed by #282

Comments

@knm3000
Copy link
Contributor

knm3000 commented Jul 25, 2022

After we had introduced big-endian support in proxy-wasm #198 and fixed
#197 the following new error started happening (regression of #198):
tcp_metadata_exchange test from proxy repo fails on big-endian platform (envoy binary crash) https://github.com/istio/proxy/blob/master/test/envoye2e/tcp_metadata_exchange/tcp_metadata_exchange_test.go

go test ./test/envoye2e/tcp_metadata_exchange -v
...

[2022-07-25 11:05:29.028][240769][critical][assert] [external/envoy/source/exe/main_common.cc:84] panic: out of memory
[2022-07-25 11:05:29.091][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3ac8100000000
[2022-07-25 11:05:29.226][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-07-25 11:05:29.226][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: b977767d8be6ce88e1956cd1f6165242aab2a572/1.20.6/Modified/RELEASE/OpenSSL
[2022-07-25 11:05:29.256][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x20025f33490]
[2022-07-25 11:05:29.257][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #1: __pthread_kill_implementation [0x2002679eb86]
[2022-07-25 11:05:29.258][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #2: gsignal [0x200267500e0]
[2022-07-25 11:05:29.258][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #3: abort [0x2002672b100]
[2022-07-25 11:05:29.362][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #4: Envoy::MainCommonBase::MainCommonBase()::$_1::__invoke() [0x2aa3cea39bc]
[2022-07-25 11:05:29.372][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #5: tcmalloc::cpp_throw_oom() [0x2aa4076b0a0]
[2022-07-25 11:05:29.381][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #6: tcmalloc::allocate_full_cpp_throw_oom() [0x2aa4076bd1c]
[2022-07-25 11:05:29.399][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #7: flatbuffers::FlatBufferBuilder::CreateString() [0x2aa3ce7140a]
[2022-07-25 11:05:29.408][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #8: Wasm::Common::extractLocalNodeFlatBuffer() [0x2aa3ce6aa66]
[2022-07-25 11:05:29.417][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #9: proxy_wasm::null_plugin::Stats::PluginRootContext::configure() [0x2aa3cdbbcc0]
[2022-07-25 11:05:29.426][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #10: proxy_wasm::null_plugin::Stats::PluginRootContext::onConfigure() [0x2aa3cdbbc18]
[2022-07-25 11:05:29.435][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #11: std::_Function_handler<>::_M_invoke() [0x2aa3d9a2898]
[2022-07-25 11:05:29.445][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #12: proxy_wasm::ContextBase::onConfigure() [0x2aa3d9a83c8]
[2022-07-25 11:05:29.463][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #13: proxy_wasm::WasmBase::configure() [0x2aa3d9c42b4]
[2022-07-25 11:05:29.474][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #14: proxy_wasm::createWasm() [0x2aa3d9c66da]
[2022-07-25 11:05:29.484][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #15: Envoy::Extensions::Common::Wasm::createWasm()::$_8::operator()() [0x2aa3d6d05f4]
[2022-07-25 11:05:29.512][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #16: Envoy::Extensions::Common::Wasm::createWasm() [0x2aa3d6cd924]
[2022-07-25 11:05:29.527][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #17: Envoy::Extensions::NetworkFilters::Wasm::FilterConfig::FilterConfig() [0x2aa3d5ecf9a]
[2022-07-25 11:05:29.537][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #18: Envoy::Extensions::NetworkFilters::Wasm::WasmFilterConfig::createFilterFactoryFromProtoTyped() [0x2aa3d5eab6c]
[2022-07-25 11:05:29.547][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #19: Envoy::Extensions::NetworkFilters::Common::FactoryBase<>::createFilterFactoryFromProto() [0x2aa3d5eb976]
[2022-07-25 11:05:29.557][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #20: Envoy::Server::ProdListenerComponentFactory::createNetworkFilterFactoryList_() [0x2aa3f96c45a]
[2022-07-25 11:05:29.575][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #21: Envoy::Server::ListenerFilterChainFactoryBuilder::buildFilterChainInternal() [0x2aa3f97985a]
[2022-07-25 11:05:29.586][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #22: Envoy::Server::ListenerFilterChainFactoryBuilder::buildFilterChain() [0x2aa3f979526]
[2022-07-25 11:05:29.599][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #23: Envoy::Server::FilterChainManagerImpl::addFilterChains() [0x2aa3f996c4c]
[2022-07-25 11:05:29.609][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #24: Envoy::Server::ListenerImpl::ListenerImpl() [0x2aa3f95f46a]
[2022-07-25 11:05:29.620][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #25: Envoy::Server::ListenerManagerImpl::addOrUpdateListenerInternal() [0x2aa3f974652]
[2022-07-25 11:05:29.630][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #26: Envoy::Server::ListenerManagerImpl::addOrUpdateListener() [0x2aa3f9735d8]
[2022-07-25 11:05:29.658][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #27: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x2aa3f9e8e16]
[2022-07-25 11:05:29.691][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #28: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x2aa3f9ea7f8]
[2022-07-25 11:05:29.718][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #29: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x2aa3fb27e70]
[2022-07-25 11:05:29.739][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #30: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x2aa3fb309a8]
[2022-07-25 11:05:29.748][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #31: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x2aa3fb34a3c]
[2022-07-25 11:05:29.763][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #32: Envoy::Grpc::AsyncStreamImpl::onData() [0x2aa3fb570c6]
[2022-07-25 11:05:29.780][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #33: Envoy::Http::AsyncStreamImpl::encodeData() [0x2aa3fb60b4c]
[2022-07-25 11:05:29.789][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #34: Envoy::Router::UpstreamRequest::decodeData() [0x2aa3fbcb9f0]
[2022-07-25 11:05:29.798][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #35: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x2aa3f81b63e]
[2022-07-25 11:05:29.807][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #36: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x2aa3faa37a0]
[2022-07-25 11:05:29.816][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #37: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_22::__invoke() [0x2aa3fab0824]
[2022-07-25 11:05:29.836][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #38: nghttp2_session_on_data_received [0x2aa3fd088e4]
[2022-07-25 11:05:29.846][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #39: nghttp2_session_mem_recv [0x2aa3fd0ab0a]
[2022-07-25 11:05:29.855][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #40: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x2aa3faa1a00]
[2022-07-25 11:05:29.864][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #41: Envoy::Http::CodecClient::onData() [0x2aa3f8cb478]
[2022-07-25 11:05:29.873][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #42: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x2aa3f8cd5e2]
[2022-07-25 11:05:29.882][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #43: Envoy::Network::FilterManagerImpl::onRead() [0x2aa3fb19de2]
[2022-07-25 11:05:29.891][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #44: Envoy::Network::ConnectionImpl::onReadReady() [0x2aa3fb0f666]
[2022-07-25 11:05:29.900][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #45: Envoy::Network::ConnectionImpl::onFileEvent() [0x2aa3fb0ccc8]
[2022-07-25 11:05:29.908][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #46: std::_Function_handler<>::_M_invoke() [0x2aa3fae79da]
[2022-07-25 11:05:29.917][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #47: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x2aa3fae8d84]
[2022-07-25 11:05:29.926][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #48: event_process_active_single_queue [0x2aa3fcf8b3e]
[2022-07-25 11:05:29.936][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #49: event_base_loop [0x2aa3fcf757e]
[2022-07-25 11:05:29.945][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #50: Envoy::Event::DispatcherImpl::run() [0x2aa3fae512e]
[2022-07-25 11:05:29.973][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #51: Envoy::Server::InstanceImpl::run() [0x2aa3f4cadf6]
[2022-07-25 11:05:29.983][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #52: Envoy::MainCommonBase::run() [0x2aa3cea1312]
[2022-07-25 11:05:29.992][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #53: Envoy::MainCommon::main() [0x2aa3cea1c76]
[2022-07-25 11:05:30.001][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #54: main [0x2aa3ce9d13e]
[2022-07-25 11:05:30.001][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #55: __libc_start_call_main [0x200267317f2]
[2022-07-25 11:05:30.002][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #56: __libc_start_main@GLIBC_2.2 [0x200267318d0]
[2022-07-25 11:05:30.010][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #57: [0x2aa3cc9ee90]
AsyncClient 0x2aa43ecb080, stream_id_: 8385127285791150716

The root cause is that tcp_metadata_exchange test uses null-vm, and in #198 we introduced byte swapping on big-endian for all wasm runtimes (including null-vm). Since null-vm is not using wasm binary, we should not reverse bytes for null-vm (for other wasm runtimes we still need to reverse bytes on big-endian).
Opened #282 to fix it.
@knm3000 knm3000 mentioned this issue Jul 25, 2022
Merged
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Aug 17, 2022
@knm3000
Do not reverse bytes for nullvm
d61f579 
Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Sep 9, 2022
@knm3000
Do not reverse bytes for nullvm
a32a30c 
additional changes:
- moved the call to isWasmByteOrder() into htowasm/wasmtoh macros
- added surrounding brackets to htowasm/wasmtoh
- removed debug leftover

Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Sep 9, 2022
@knm3000
Do not reverse bytes for nullvm
6e273eb 
additional changes:
- moved the call to isWasmByteOrder() into htowasm/wasmtoh macros
- added surrounding brackets to htowasm/wasmtoh
- removed debug leftover

Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Sep 14, 2022
@knm3000
Do not reverse bytes for nullvm
ad11d07 
more changes:
- renamed is_wasm_byte_order to vm_uses_wasm_byte_order
- renamed isWasmByteOrder() to usesWasmByteOrder()

Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Sep 28, 2022
@knm3000
Do not reverse bytes for nullvm
dec6b94 
- use buf and buf_len variables for readability
- added back word.h include
- hardcoded usesWasmByteOrder() to true

Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Sep 30, 2022
@knm3000
Do not reverse bytes for nullvm
65c8c8f 
- moved pairs test into a new file test/pairs_util_test.cc
- run pairs test on all platforms
- made contextOrEffectiveContext() null check a bit more readable

Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Sep 30, 2022
@knm3000
Do not reverse bytes for nullvm
d0a3249 
- moved pairs test into a new file test/pairs_util_test.cc
- run pairs test on all platforms
- made contextOrEffectiveContext() null check a bit more readable

Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
knm3000 added a commit to knm3000/proxy-wasm-cpp-host that referenced this issue Oct 1, 2022
@knm3000
Do not reverse bytes for nullvm
a82cc8c 
more changes:
- removed unnecessary #include from test/null_vm_test.cc

Fixes proxy-wasm#294

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
PiotrSikora pushed a commit that referenced this issue Oct 2, 2022
@knm3000
Do not reverse bytes for NullVM. (#282)
537b944 
Fixes #294.

Signed-off-by: Konstantin Maksimov <konstantin.maksimov@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Do not reverse bytes for NullVM. knm3000/proxy-wasm-cpp-host
1 participant
@knm3000