Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid null dereference after "Unknown packet X from server" #8071

Merged

Conversation

azat
Copy link
Collaborator

@azat azat commented Dec 6, 2019

Changelog category (leave one):

  • Bug Fix

Changelog entry (up to few sentences, required except for Non-significant/Documentation categories):
Avoid null dereference after "Unknown packet X from server"

Detailed description (optional):

stracktrace:

  ...
  2  DB::WriteBuffer::nextIfAtEnd (this=0x0) at ../dbms/src/IO/WriteBuffer.h:66
  3  DB::writeVarUInt (ostr=..., x=3) at ../dbms/src/IO/VarInt.h:191
  4  DB::Connection::sendCancel (this=0x7f0b7f5e5610) at ../dbms/src/Client/Connection.cpp:444
  5  0x0000561738d0b565 in DB::MultiplexedConnections::sendCancel (this=0x7f0c9b0173e0) at ../dbms/src/Client/MultiplexedConnections.cpp:174
  6  0x00005617387d22df in DB::RemoteBlockInputStream::tryCancel (this=0x7f0cc45f9810, reason=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:352
  7  0x00005617387d2a37 in DB::RemoteBlockInputStream::cancel (this=<optimized out>, kill=false) at ../dbms/src/DataStreams/RemoteBlockInputStream.cpp:121
  8  0x000056173891f10c in DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::cancel (this=0x7f0c370ec168, kill=kill@entry=false)
  9  0x000056173892cfce in DB::UnionBlockInputStream::cancel (kill=false, this=<optimized out>) at ../dbms/src/DataStreams/UnionBlockInputStream.h:99
  10 DB::UnionBlockInputStream::Handler::onException (this=0x7f0c370ec160, exception=...) at ../dbms/src/DataStreams/UnionBlockInputStream.h:240
  11 DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread (this=0x7f0c370ec168, thread_group=..., thread_num=<optimized out>)
      at ../dbms/src/DataStreams/ParallelInputsProcessor.h:217

And in onException frame there is:

  (gdb) p ((DB::Exception *)exception._M_exception_object)._msg._M_dataplus._M_p
  $6 = (...) 0x7f0c9b1e7a40 "Unknown packet 9 from server HOSTNAME:PORT"

And on "Unknown packet" there is disconnect() before, see default in
switch statement at Connection::receivePacket().

stracktrace:
  ...
  2  DB::WriteBuffer::nextIfAtEnd (this=0x0) at ../dbms/src/IO/WriteBuffer.h:66
  3  DB::writeVarUInt (ostr=..., x=3) at ../dbms/src/IO/VarInt.h:191
  4  DB::Connection::sendCancel (this=0x7f0b7f5e5610) at ../dbms/src/Client/Connection.cpp:444
  5  0x0000561738d0b565 in DB::MultiplexedConnections::sendCancel (this=0x7f0c9b0173e0) at ../dbms/src/Client/MultiplexedConnections.cpp:174
  6  0x00005617387d22df in DB::RemoteBlockInputStream::tryCancel (this=0x7f0cc45f9810, reason=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:352
  7  0x00005617387d2a37 in DB::RemoteBlockInputStream::cancel (this=<optimized out>, kill=false) at ../dbms/src/DataStreams/RemoteBlockInputStream.cpp:121
  8  0x000056173891f10c in DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::cancel (this=0x7f0c370ec168, kill=kill@entry=false)
  9  0x000056173892cfce in DB::UnionBlockInputStream::cancel (kill=false, this=<optimized out>) at ../dbms/src/DataStreams/UnionBlockInputStream.h:99
  10 DB::UnionBlockInputStream::Handler::onException (this=0x7f0c370ec160, exception=...) at ../dbms/src/DataStreams/UnionBlockInputStream.h:240
  11 DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread (this=0x7f0c370ec168, thread_group=..., thread_num=<optimized out>)
      at ../dbms/src/DataStreams/ParallelInputsProcessor.h:217

And in onException frame there is:
  (gdb) p ((DB::Exception *)exception._M_exception_object)._msg._M_dataplus._M_p
  $6 = (...) 0x7f0c9b1e7a40 "Unknown packet 9 from server HOSTNAME:PORT"

And on "Unknown packet" there is disconnect() before, see default in
switch statement at Connection::receivePacket().
Copy link
Member

@alexey-milovidov alexey-milovidov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

There is no test, but it will be difficult to provide (a fake server). So, no need for test.

@alexey-milovidov alexey-milovidov merged commit bef3a0d into ClickHouse:master Dec 6, 2019
@azat azat deleted the Connection-sendCancel-SIGSEGV branch December 6, 2019 19:18
@KochetovNicolai KochetovNicolai added the pr-bugfix Pull request with bugfix, not backported by default label Dec 16, 2019
KochetovNicolai pushed a commit that referenced this pull request Dec 17, 2019
Avoid null dereference after "Unknown packet X from server"

(cherry picked from commit bef3a0d)
KochetovNicolai pushed a commit that referenced this pull request Dec 19, 2019
Avoid null dereference after "Unknown packet X from server"

(cherry picked from commit bef3a0d)
KochetovNicolai pushed a commit that referenced this pull request Dec 20, 2019
Avoid null dereference after "Unknown packet X from server"

(cherry picked from commit bef3a0d)
vitlibar pushed a commit that referenced this pull request Dec 26, 2019
Avoid null dereference after "Unknown packet X from server"

(cherry picked from commit bef3a0d)
vitlibar pushed a commit that referenced this pull request Dec 27, 2019
Avoid null dereference after "Unknown packet X from server"

(cherry picked from commit bef3a0d)
@filimonov
Copy link
Contributor

One more symptom of the same bug reported:

2020.07.03 10:41:25.431186 [ 450 ] {} <Fatal> BaseDaemon: ########################################
2020.07.03 10:41:25.431247 [ 450 ] {} <Fatal> BaseDaemon: (version 19.15.3.6) (from thread 305) Received signal Segmentation fault (11).
2020.07.03 10:41:25.431263 [ 450 ] {} <Fatal> BaseDaemon: Address: 0x8 Access: read. Address not mapped to object.
2020.07.03 10:41:25.431283 [ 450 ] {} <Fatal> BaseDaemon: Stack trace: 0x63d5601 0x63e710d 0x5f1c9c4 0x5f1d041 0x65dafdc 0x65dc1fb 0x63140bd 0x63140bd 0x33dd247 0x60f30cc 0x33dd247 0x326135c 0x34b70b1 0x34ba0f1 0x6bd3759 0x6bd03bf 0x6bd0ab5 0x7055f91 0x7051d18 0x773411f 0x7f68fa13ee65 0x7f68f9a5b88d
2020.07.03 10:41:25.431327 [ 450 ] {} <Fatal> BaseDaemon: 3. 0x63d5601 DB::Connection::sendCancel() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431347 [ 450 ] {} <Fatal> BaseDaemon: 4. 0x63e710d DB::MultiplexedConnections::sendCancel() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431377 [ 450 ] {} <Fatal> BaseDaemon: 5. 0x5f1c9c4 DB::RemoteBlockInputStream::tryCancel(char const*) /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431400 [ 450 ] {} <Fatal> BaseDaemon: 6. 0x5f1d041 DB::RemoteBlockInputStream::cancel(bool) /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431423 [ 450 ] {} <Fatal> BaseDaemon: 7. 0x65dafdc DB::MergingAggregatedMemoryEfficientBlockInputStream::cancel(bool) /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431441 [ 450 ] {} <Fatal> BaseDaemon: 8. 0x65dc1fb DB::MergingAggregatedMemoryEfficientBlockInputStream::~MergingAggregatedMemoryEfficientBlockInputStream() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431462 [ 450 ] {} <Fatal> BaseDaemon: 9. 0x63140bd DB::ExpressionBlockInputStream::~ExpressionBlockInputStream() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431478 [ 450 ] {} <Fatal> BaseDaemon: 10. 0x63140bd DB::ExpressionBlockInputStream::~ExpressionBlockInputStream() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431499 [ 450 ] {} <Fatal> BaseDaemon: 11. 0x33dd247 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431519 [ 450 ] {} <Fatal> BaseDaemon: 12. 0x60f30cc DB::ProcessListEntry::~ProcessListEntry() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431534 [ 450 ] {} <Fatal> BaseDaemon: 13. 0x33dd247 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431553 [ 450 ] {} <Fatal> BaseDaemon: 14. 0x326135c ? /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431585 [ 450 ] {} <Fatal> BaseDaemon: 15. 0x34b70b1 DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&) /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431603 [ 450 ] {} <Fatal> BaseDaemon: 16. 0x34ba0f1 DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431624 [ 450 ] {} <Fatal> BaseDaemon: 17. 0x6bd3759 Poco::Net::HTTPServerConnection::run() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431640 [ 450 ] {} <Fatal> BaseDaemon: 18. 0x6bd03bf Poco::Net::TCPServerConnection::start() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431655 [ 450 ] {} <Fatal> BaseDaemon: 19. 0x6bd0ab5 Poco::Net::TCPServerDispatcher::run() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431673 [ 450 ] {} <Fatal> BaseDaemon: 20. 0x7055f91 Poco::PooledThread::run() /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431689 [ 450 ] {} <Fatal> BaseDaemon: 21. 0x7051d18 Poco::ThreadImpl::runnableEntry(void*) /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431706 [ 450 ] {} <Fatal> BaseDaemon: 22. 0x773411f ? /export/ica/clickhouse/clickhouse
2020.07.03 10:41:25.431733 [ 450 ] {} <Fatal> BaseDaemon: 23. 0x7f68fa13ee65 start_thread /usr/lib64/libpthread-2.17.so
2020.07.03 10:41:25.431754 [ 450 ] {} <Fatal> BaseDaemon: 24. 0x7f68f9a5b88d clone /usr/lib64/libc-2.17.so

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-bugfix Pull request with bugfix, not backported by default
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants