Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Read parquet file panic #108

Closed
BohuTANG opened this issue Mar 6, 2022 · 3 comments
Closed

Read parquet file panic #108

BohuTANG opened this issue Mar 6, 2022 · 3 comments
Assignees
@Xuanwo
Labels
enhancement New feature or request

Comments

@BohuTANG
Copy link

BohuTANG commented Mar 6, 2022

How to reproduce

opendal: opendal-0.1.4

branch:
databendlabs/databend#4328

1. Run minio:

docker run -d -p 9900:9000 --name minio \
                -e "MINIO_ACCESS_KEY=minioadmin" \
                -e "MINIO_SECRET_KEY=minioadmin" \
                -v /tmp/data:/data \
                -v /tmp/config:/root/.minio \
                minio/minio server /data

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_EC2_METADATA_DISABLED=true
aws --endpoint-url http://127.0.0.1:9900/ s3 mb s3://testbucket
aws --endpoint-url http://127.0.0.1:9900/ s3 cp tests/data s3://testbucket/admin/data  --recursive

2. Start databend-query and create ontime table

https://repo.databend.rs/t_ontime/create_table.sql

3. SQL

copy into ontime from 's3://testbucket/admin/data/ontime_200.parquet' credentials=(aws_key_id='minioadmin' aws_secret_key='minioadmin') FILE_FORMAT = (type = 'parquet');

Server logs:

2022-03-06T00:19:16.316335Z ERROR common_tracing::panic_hook: panicked at '`async fn` resumed after completion', /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/opendal-0.1.4/src/io.rs:147:45 backtrace=   0: common_tracing::panic_hook::set_panic_hook::{{closure}}
             at common/tracing/src/panic_hook.rs:25:25
   1: std::panicking::rust_panic_with_hook
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:586:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/panicking.rs:143:14
   6: core::panicking::panic
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/panicking.rs:48:5
   7: <opendal::io::Reader as futures_io::if_std::AsyncSeek>::poll_seek::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/opendal-0.1.4/src/io.rs:147:45
   8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/future.rs:124:9
  10: <opendal::io::Reader as futures_io::if_std::AsyncSeek>::poll_seek
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/opendal-0.1.4/src/io.rs:129:26
  11: <&mut T as futures_io::if_std::AsyncSeek>::poll_seek
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-io-0.3.21/src/lib.rs:479:17
  12: <futures_util::io::seek::Seek<S> as core::future::future::Future>::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/io/seek.rs:28:9
  13: parquet2::read::stream::stream_len::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/parquet2-0.10.2/src/read/stream.rs:14:50
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  15: parquet2::read::stream::read_metadata::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/parquet2-0.10.2/src/read/stream.rs:30:39
  16: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  17: arrow2::io::parquet::read::read_metadata_async::{{closure}}
             at /home/bohu/.cargo/git/checkouts/arrow2-6249446b5f8db6f7/f71124b/src/io/parquet/read/mod.rs:68:36
  18: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  19: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.31/src/instrument.rs:272:9
  20: <common_streams::sources::source_parquet::ParquetSource<R> as common_streams::sources::source::Source>::read::{{closure}}::{{closure}}
             at common/streams/src/sources/source_parquet.rs:102:73
  21: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  22: <common_streams::sources::source_parquet::ParquetSource<R> as common_streams::sources::source::Source>::read::{{closure}}
             at common/streams/src/sources/source_parquet.rs:95:5
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  24: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/future.rs:124:9
  25: <databend_query::storages::s3::s3_external_source::ExternalSource as databend_query::pipelines::new::processors::sources::async_source::AsyncSource>::generate::{{closure}}
             at query/src/storages/s3/s3_external_source.rs:181:52
  26: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  27: <databend_query::pipelines::new::processors::sources::async_source::AsyncSourcer<T> as databend_query::pipelines::new::processors::processor::Processor>::async_process::{{closure}}
             at query/src/pipelines/new/processors/sources/async_source.rs:86:36
  28: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  29: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/future.rs:124:9
  30: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/future.rs:124:9
  31: databend_query::pipelines::new::executor::executor_worker_context::ExecutorWorkerContext::execute_async_task::{{closure}}
             at query/src/pipelines/new/executor/executor_worker_context.rs:90:48
  32: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/future/mod.rs:91:19
  33: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
  34: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
  35: tokio::runtime::task::core::CoreStage<T>::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
  36: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
  37: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/panic/unwind_safe.rs:271:9
  38: std::panicking::try::do_call
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:492:40
  39: __rust_try
  40: std::panicking::try
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:456:19
  41: std::panic::catch_unwind
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panic.rs:137:14
  42: tokio::runtime::task::harness::poll_future
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
  43: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
  44: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
  45: tokio::runtime::task::raw::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
  46: tokio::runtime::task::raw::RawTask::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
  47: tokio::runtime::task::LocalNotified<S>::run
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:347:9
  48: tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:425:13
  49: tokio::coop::with_budget::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
  50: std::thread::local::LocalKey<T>::try_with
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/thread/local.rs:413:16
  51: std::thread::local::LocalKey<T>::with
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/thread/local.rs:389:9
  52: tokio::coop::with_budget
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
      tokio::coop::budget
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
      tokio::runtime::thread_pool::worker::Context::run_task
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:424:9
  53: tokio::runtime::thread_pool::worker::Context::run
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:391:24
  54: tokio::runtime::thread_pool::worker::run::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:376:17
  55: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
  56: tokio::runtime::thread_pool::worker::run
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:373:5
  57: tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:352:45
  58: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/task.rs:42:21
  59: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
  60: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
  61: tokio::runtime::task::core::CoreStage<T>::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
  62: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
  63: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/panic/unwind_safe.rs:271:9
  64: std::panicking::try::do_call
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:492:40
  65: __rust_try
  66: std::panicking::try
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:456:19
  67: std::panic::catch_unwind
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panic.rs:137:14
  68: tokio::runtime::task::harness::poll_future
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
  69: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
  70: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
  71: tokio::runtime::task::raw::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
  72: tokio::runtime::task::raw::RawTask::poll
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
  73: tokio::runtime::task::UnownedTask<S>::run
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:384:9
  74: tokio::runtime::blocking::pool::Task::run
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:91:9
  75: tokio::runtime::blocking::pool::Inner::run
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:308:17
  76: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:288:17
  77: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/sys_common/backtrace.rs:122:18
  78: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/thread/mod.rs:498:17
  79: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/panic/unwind_safe.rs:271:9
  80: std::panicking::try::do_call
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:492:40
  81: __rust_try
  82: std::panicking::try
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panicking.rs:456:19
  83: std::panic::catch_unwind
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/panic.rs:137:14
  84: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/thread/mod.rs:497:30
  85: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/core/src/ops/function.rs:227:5
  86: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/alloc/src/boxed.rs:1854:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/alloc/src/boxed.rs:1854:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/4ce3749235fc31d15ebd444b038a9877e8c700d7/library/std/src/sys/unix/thread.rs:108:17
  87: start_thread
             at ./nptl/./nptl/pthread_create.c:442:8
  88: __GI___clone3
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 panic.file="/home/bohu/.cargo/registry/src/github.com-1ecc6299db9ec823/opendal-0.1.4/src/io.rs" panic.line=147 panic.column=45
@BohuTANG BohuTANG added the bug Something isn't working label Mar 6, 2022
@BohuTANG
Copy link
Author

BohuTANG commented Mar 6, 2022

This caused by parquet file crate panic.
Is it possible that we handle the panic from the underlying crate in opendal?

@BohuTANG BohuTANG added enhancement New feature or request and removed bug Something isn't working labels Mar 6, 2022
@BohuTANG
Copy link
Author

BohuTANG commented Mar 7, 2022

I think this is in very low priority, possibly a robustness problem.

@Xuanwo
Copy link
Member

Xuanwo commented May 19, 2022

Not handled by opendal, closing now.

@Xuanwo Xuanwo closed this as completed May 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Xuanwo Xuanwo

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@BohuTANG @Xuanwo