Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: CREATE SINK panic #8482

Closed
JuchangGit opened this issue Mar 11, 2023 · 13 comments
Closed

bug: CREATE SINK panic #8482

JuchangGit opened this issue Mar 11, 2023 · 13 comments
Assignees
@tabVersion
Labels
type/bug Something isn't working
Milestone
release-0.19

Comments

@JuchangGit
Copy link

JuchangGit commented Mar 11, 2023 •
edited by fuyufjh
Loading

ERROR
image

ENV
image
image

Error message

ERROR:  QueryError: internal error: Rpc error: gRPC error (Internal error): Actor 72 exit unexpectedly: failed to send message to actor 120: Barrier(Barrier { epoch: EpochPair { curr: 4015336712372224, prev: 4015336676196352 }, mutation: Some(Add { adds: {66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [114] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [113] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [119] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [115] }], 68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [116] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [120] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [118] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 15, downstream_actor_id: [117] }]}, splits: {} }), checkpoint: true, passed_actors: [72] })
  backtrace of `StreamError`:
   0: <risingwave_stream::error::StreamError as core::convert::From<risingwave_stream::error::Inner>>::from
             at ./risingwave/src/stream/src/error.rs:47:10
   1: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   2: <risingwave_stream::error::StreamError as core::convert::From<anyhow::Error>>::from
             at ./risingwave/src/stream/src/error.rs:110:9
   3: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   4: <risingwave_stream::executor::exchange::output::LocalOutput as risingwave_stream::executor::exchange::output::Output>::send::{{closure}}::{{closure}}
             at ./risingwave/src/stream/src/executor/exchange/output.rs:81:17
   5: core::result::Result<T,E>::map_err
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/result.rs:860:27
   6: <risingwave_stream::executor::exchange::output::LocalOutput as risingwave_stream::executor::exchange::output::Output>::send::{{closure}}
             at ./risingwave/src/stream/src/executor/exchange/output.rs:76:9
   7: <core::pin::Pin<P> as core::future::future::Future>::poll
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/future/future.rs:125:9
   8: <risingwave_stream::executor::dispatch::SimpleDispatcher as risingwave_stream::executor::dispatch::Dispatcher>::dispatch_barrier::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:806:63
   9: risingwave_stream::executor::dispatch::DispatcherImpl::dispatch_barrier::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:354:85
  10: risingwave_stream::executor::dispatch::DispatchExecutorInner::dispatch::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:94:65
  11: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
  12: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:279:22
  13: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.6/src/lib.rs:463:33
  14: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.26/src/stream.rs:120:9
  15: <&mut S as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.26/src/stream.rs:104:9
  16: <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-stream-0.1.11/src/stream_ext/next.rs:42:9
  17: <minitrace::future::InSpan<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/minitrace-0.4.0/src/future.rs:118:19
  18: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
             at ./risingwave/src/utils/async_stack_trace/src/lib.rs:182:23
  19: risingwave_stream::executor::actor::Actor<C>::run_consumer::{{closure}}
             at ./risingwave/src/stream/src/executor/actor.rs:164:13
  20: <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/maybe_done.rs:68:48
  21: risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/join.rs:126:24
  22: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  23: risingwave_stream::executor::actor::Actor<C>::run::{{closure}}
             at ./risingwave/src/stream/src/executor/actor.rs:131:9
  24: risingwave_stream::task::stream_manager::LocalStreamManagerCore::build_actors::{{closure}}::{{closure}}
             at ./risingwave/src/stream/src/task/stream_manager.rs:653:50
  25: async_stack_trace::manager::TraceReporter::trace::{{closure}}::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/select.rs:517:49
  26: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  27: async_stack_trace::manager::TraceReporter::trace::{{closure}}::{{closure}}
             at ./risingwave/src/utils/async_stack_trace/src/manager.rs:124:21
  28: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:348:35
  29: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:233:19
  30: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:345:13
  31: async_stack_trace::manager::TraceReporter::trace::{{closure}}
             at ./risingwave/src/utils/async_stack_trace/src/manager.rs:131:13
  32: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.26/src/future/either.rs:109:32
  33: <tokio_metrics::task::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-metrics-0.1.0/src/task.rs:2365:19
  34: task_stats_alloc::allocation_stat::{{closure}}::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/select.rs:517:49
  35: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  36: task_stats_alloc::allocation_stat::{{closure}}::{{closure}}
             at ./risingwave/src/utils/task_stats_alloc/src/lib.rs:124:26
  37: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:348:35
  38: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:233:19
  39: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:345:13
  40: task_stats_alloc::allocation_stat::{{closure}}
             at ./risingwave/src/utils/task_stats_alloc/src/lib.rs:131:9
  41: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  42: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  43: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  44: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  45: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  46: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  47: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  48: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  49: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  50: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  51: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  52: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  53: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:394:9
  55: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:487:21
  56: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
  57: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
  58: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  59: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  60: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  61: tokio::macros::scoped_tls::ScopedKey<T>::set
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/scoped_tls.rs:61:9
  62: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  63: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  64: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/task.rs:42:21
  65: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  66: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  67: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  68: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  69: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  70: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  71: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  72: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  73: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  74: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  75: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  76: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  77: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  78: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:431:9
  79: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:159:9
  80: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:511:17
  81: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:469:13
  82: std::sys_common::backtrace::__rust_begin_short_backtrace
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys_common/backtrace.rs:121:18
  83: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:558:17
  84: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  85: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  86: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  87: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  88: std::thread::Builder::spawn_unchecked_::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:557:30
  89: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/ops/function.rs:250:5
  90: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  91: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  92: std::sys::unix::thread::Thread::new::thread_start
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys/unix/thread.rs:108:17
  93: <unknown>
  94: __clone
@github-actions github-actions bot added this to the release-0.1.18 milestone Mar 11, 2023
@xxchan
Copy link
Member

xxchan commented Mar 11, 2023 •
edited
Loading

Thanks for reporting the issue. Could you provide the log of the connector node?

@JuchangGit
Copy link
Author

image

@JuchangGit
Copy link
Author

Thanks for reporting the issue. Could you provide the log of the connector node?

There doesn't seem to be anything useful in the log

@JuchangGit
Copy link
Author

I just tested sink kafka and there is no problem

@BugenZhao
Copy link
Member

Could you please also provide the log of meta node and compute node?

@JuchangGit
Copy link
Author

JuchangGit commented Mar 12, 2023 •
edited by fuyufjh
Loading

compute-node-0 logs

2023-03-12T08:00:58.568616Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: update to hummock version: 22909, epoch: 4022131273629696
2023-03-12T08:00:58.683490Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 215
2023-03-12T08:00:58.683490Z ERROR risingwave_stream::task::stream_manager: actor exit actor=71 error=failed to send message to actor 159: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [71] })
2023-03-12T08:00:58.683701Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 216
2023-03-12T08:00:58.683703Z ERROR risingwave_stream::task::stream_manager: actor exit actor=72 error=failed to send message to actor 160: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [72] })
2023-03-12T08:00:58.683751Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 214
2023-03-12T08:00:58.683750Z ERROR risingwave_stream::task::stream_manager: actor exit actor=70 error=failed to send message to actor 158: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [70] })
2023-03-12T08:00:58.683827Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 213
2023-03-12T08:00:58.683825Z ERROR risingwave_stream::task::stream_manager: actor exit actor=69 error=failed to send message to actor 157: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [69] })
2023-03-12T08:00:58.683906Z ERROR risingwave_stream::task::stream_manager: actor exit actor=68 error=failed to send message to actor 156: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [68] })
2023-03-12T08:00:58.683906Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 212
2023-03-12T08:00:58.684004Z ERROR risingwave_stream::task::stream_manager: actor exit actor=65 error=failed to send message to actor 153: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [65] })
2023-03-12T08:00:58.684006Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 209
2023-03-12T08:00:58.684030Z ERROR risingwave_stream::task::stream_manager: actor exit actor=66 error=failed to send message to actor 154: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [66] })
2023-03-12T08:00:58.684032Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 210
2023-03-12T08:00:58.684063Z ERROR risingwave_stream::task::stream_manager: actor exit actor=67 error=failed to send message to actor 155: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [67] })
2023-03-12T08:00:58.684064Z  INFO risingwave_storage::hummock::event_handler::hummock_event_handler: read version deregister: table_id: 1015, instance_id: 211
2023-03-12T08:00:58.684118Z ERROR risingwave_compute::rpc::service::stream_service: failed to collect barrier: Actor 71 exit unexpectedly: failed to send message to actor 159: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [71] })
  backtrace of `StreamError`:
   0: <risingwave_stream::error::StreamError as core::convert::From<risingwave_stream::error::Inner>>::from
             at ./risingwave/src/stream/src/error.rs:47:10
   1: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   2: <risingwave_stream::error::StreamError as core::convert::From<anyhow::Error>>::from
             at ./risingwave/src/stream/src/error.rs:110:9
   3: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   4: <risingwave_stream::executor::exchange::output::LocalOutput as risingwave_stream::executor::exchange::output::Output>::send::{{closure}}::{{closure}}
             at ./risingwave/src/stream/src/executor/exchange/output.rs:81:17
   5: core::result::Result<T,E>::map_err
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/result.rs:860:27
   6: <risingwave_stream::executor::exchange::output::LocalOutput as risingwave_stream::executor::exchange::output::Output>::send::{{closure}}
             at ./risingwave/src/stream/src/executor/exchange/output.rs:76:9
   7: <core::pin::Pin<P> as core::future::future::Future>::poll
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/future/future.rs:125:9
   8: <risingwave_stream::executor::dispatch::SimpleDispatcher as risingwave_stream::executor::dispatch::Dispatcher>::dispatch_barrier::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:806:63
   9: risingwave_stream::executor::dispatch::DispatcherImpl::dispatch_barrier::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:354:85
  10: risingwave_stream::executor::dispatch::DispatchExecutorInner::dispatch::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:94:65
  11: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
  12: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:279:22
  13: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.6/src/lib.rs:463:33
  14: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.26/src/stream.rs:120:9
  15: <&mut S as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.26/src/stream.rs:104:9
  16: <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-stream-0.1.11/src/stream_ext/next.rs:42:9
  17: <minitrace::future::InSpan<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/minitrace-0.4.0/src/future.rs:118:19
  18: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
             at ./risingwave/src/utils/async_stack_trace/src/lib.rs:182:23
  19: risingwave_stream::executor::actor::Actor<C>::run_consumer::{{closure}}
             at ./risingwave/src/stream/src/executor/actor.rs:164:13
  20: <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/maybe_done.rs:68:48
  21: risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/join.rs:126:24
  22: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  23: risingwave_stream::executor::actor::Actor<C>::run::{{closure}}
             at ./risingwave/src/stream/src/executor/actor.rs:131:9
  24: risingwave_stream::task::stream_manager::LocalStreamManagerCore::build_actors::{{closure}}::{{closure}}
             at ./risingwave/src/stream/src/task/stream_manager.rs:653:50
  25: async_stack_trace::manager::TraceReporter::trace::{{closure}}::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/select.rs:517:49
  26: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  27: async_stack_trace::manager::TraceReporter::trace::{{closure}}::{{closure}}
             at ./risingwave/src/utils/async_stack_trace/src/manager.rs:124:21
  28: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:348:35
  29: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:233:19
  30: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:345:13
  31: async_stack_trace::manager::TraceReporter::trace::{{closure}}
             at ./risingwave/src/utils/async_stack_trace/src/manager.rs:131:13
  32: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.26/src/future/either.rs:109:32
  33: <tokio_metrics::task::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-metrics-0.1.0/src/task.rs:2365:19
  34: task_stats_alloc::allocation_stat::{{closure}}::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/select.rs:517:49
  35: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  36: task_stats_alloc::allocation_stat::{{closure}}::{{closure}}
             at ./risingwave/src/utils/task_stats_alloc/src/lib.rs:124:26
  37: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:348:35
  38: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:233:19
  39: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:345:13
  40: task_stats_alloc::allocation_stat::{{closure}}
             at ./risingwave/src/utils/task_stats_alloc/src/lib.rs:131:9
  41: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  42: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  43: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  44: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  45: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  46: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  47: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  48: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  49: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  50: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  51: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  52: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  53: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:394:9
  55: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:464:13
  56: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
  57: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
  58: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  59: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:433:24
  60: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  61: tokio::macros::scoped_tls::ScopedKey<T>::set
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/scoped_tls.rs:61:9
  62: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  63: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  64: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/task.rs:42:21
  65: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  66: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  67: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  68: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  69: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  70: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  71: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  72: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  73: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  74: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  75: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  76: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  77: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  78: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:431:9
  79: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:159:9
  80: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:511:17
  81: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:469:13
  82: std::sys_common::backtrace::__rust_begin_short_backtrace
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys_common/backtrace.rs:121:18
  83: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:558:17
  84: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  85: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  86: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  87: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  88: std::thread::Builder::spawn_unchecked_::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:557:30
  89: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/ops/function.rs:250:5
  90: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  91: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  92: std::sys::unix::thread::Thread::new::thread_start
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys/unix/thread.rs:108:17
  93: <unknown>
  94: __clone

@JuchangGit
Copy link
Author

JuchangGit commented Mar 12, 2023 •
edited by fuyufjh
Loading

meta-node-0 logs

2023-03-12T08:00:58.684650Z  WARN risingwave_meta::barrier: Failed to complete epoch 4022131601309696: Rpc error: gRPC error (Internal error): Actor 71 exit unexpectedly: failed to send message to actor 159: Barrier(Barrier { epoch: EpochPair { curr: 4022131662520320, prev: 4022131601309696 }, mutation: Some(Add { adds: {68: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [156] }], 72: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [160] }], 65: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [153] }], 71: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [159] }], 70: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [158] }], 69: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [157] }], 66: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [154] }], 67: [Dispatcher { r#type: NoShuffle, column_indices: [], hash_mapping: None, dispatcher_id: 20, downstream_actor_id: [155] }]}, splits: {} }), checkpoint: true, passed_actors: [71] })
  backtrace of `StreamError`:
   0: <risingwave_stream::error::StreamError as core::convert::From<risingwave_stream::error::Inner>>::from
             at ./risingwave/src/stream/src/error.rs:47:10
   1: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   2: <risingwave_stream::error::StreamError as core::convert::From<anyhow::Error>>::from
             at ./risingwave/src/stream/src/error.rs:110:9
   3: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   4: <risingwave_stream::executor::exchange::output::LocalOutput as risingwave_stream::executor::exchange::output::Output>::send::{{closure}}::{{closure}}
             at ./risingwave/src/stream/src/executor/exchange/output.rs:81:17
   5: core::result::Result<T,E>::map_err
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/result.rs:860:27
   6: <risingwave_stream::executor::exchange::output::LocalOutput as risingwave_stream::executor::exchange::output::Output>::send::{{closure}}
             at ./risingwave/src/stream/src/executor/exchange/output.rs:76:9
   7: <core::pin::Pin<P> as core::future::future::Future>::poll
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/future/future.rs:125:9
   8: <risingwave_stream::executor::dispatch::SimpleDispatcher as risingwave_stream::executor::dispatch::Dispatcher>::dispatch_barrier::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:806:63
   9: risingwave_stream::executor::dispatch::DispatcherImpl::dispatch_barrier::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:354:85
  10: risingwave_stream::executor::dispatch::DispatchExecutorInner::dispatch::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:94:65
  11: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
  12: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
             at ./risingwave/src/stream/src/executor/dispatch.rs:279:22
  13: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.6/src/lib.rs:463:33
  14: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.26/src/stream.rs:120:9
  15: <&mut S as futures_core::stream::Stream>::poll_next
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.26/src/stream.rs:104:9
  16: <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-stream-0.1.11/src/stream_ext/next.rs:42:9
  17: <minitrace::future::InSpan<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/minitrace-0.4.0/src/future.rs:118:19
  18: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
             at ./risingwave/src/utils/async_stack_trace/src/lib.rs:182:23
  19: risingwave_stream::executor::actor::Actor<C>::run_consumer::{{closure}}
             at ./risingwave/src/stream/src/executor/actor.rs:164:13
  20: <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/maybe_done.rs:68:48
  21: risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/join.rs:126:24
  22: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  23: risingwave_stream::executor::actor::Actor<C>::run::{{closure}}
             at ./risingwave/src/stream/src/executor/actor.rs:131:9
  24: risingwave_stream::task::stream_manager::LocalStreamManagerCore::build_actors::{{closure}}::{{closure}}
             at ./risingwave/src/stream/src/task/stream_manager.rs:653:50
  25: async_stack_trace::manager::TraceReporter::trace::{{closure}}::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/select.rs:517:49
  26: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  27: async_stack_trace::manager::TraceReporter::trace::{{closure}}::{{closure}}
             at ./risingwave/src/utils/async_stack_trace/src/manager.rs:124:21
  28: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:348:35
  29: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:233:19
  30: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:345:13
  31: async_stack_trace::manager::TraceReporter::trace::{{closure}}
             at ./risingwave/src/utils/async_stack_trace/src/manager.rs:131:13
  32: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.26/src/future/either.rs:109:32
  33: <tokio_metrics::task::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-metrics-0.1.0/src/task.rs:2365:19
  34: task_stats_alloc::allocation_stat::{{closure}}::{{closure}}::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/select.rs:517:49
  35: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/future/poll_fn.rs:58:9
  36: task_stats_alloc::allocation_stat::{{closure}}::{{closure}}
             at ./risingwave/src/utils/task_stats_alloc/src/lib.rs:124:26
  37: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:348:35
  38: tokio::task::task_local::LocalKey<T>::scope_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:233:19
  39: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/task/task_local.rs:345:13
  40: task_stats_alloc::allocation_stat::{{closure}}
             at ./risingwave/src/utils/task_stats_alloc/src/lib.rs:131:9
  41: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  42: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  43: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  44: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  45: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  46: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  47: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  48: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  49: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  50: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  51: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  52: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  53: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:394:9
  55: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:464:13
  56: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
  57: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
  58: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  59: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:433:24
  60: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  61: tokio::macros::scoped_tls::ScopedKey<T>::set
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/scoped_tls.rs:61:9
  62: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  63: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  64: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/task.rs:42:21
  65: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  66: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  67: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  68: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  69: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  70: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  71: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  72: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  73: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  74: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  75: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  76: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  77: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  78: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:431:9
  79: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:159:9
  80: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:511:17
  81: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:469:13
  82: std::sys_common::backtrace::__rust_begin_short_backtrace
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys_common/backtrace.rs:121:18
  83: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:558:17
  84: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  85: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  86: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  87: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  88: std::thread::Builder::spawn_unchecked_::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:557:30
  89: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/ops/function.rs:250:5
  90: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  91: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  92: std::sys::unix::thread::Thread::new::thread_start
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys/unix/thread.rs:108:17
  93: <unknown>
  94: __clone

  backtrace of `MetaError`:
   0: <risingwave_meta::error::MetaError as core::convert::From<risingwave_meta::error::MetaErrorInner>>::from
             at ./risingwave/src/meta/src/error.rs:75:33
   1: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   2: <risingwave_meta::error::MetaError as core::convert::From<risingwave_rpc_client::error::RpcError>>::from
             at ./risingwave/src/meta/src/error.rs:158:9
   3: <T as core::convert::Into<U>>::into
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   4: core::ops::function::FnOnce::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/ops/function.rs:250:5
   5: core::result::Result<T,E>::map_err
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/result.rs:860:27
   6: risingwave_meta::barrier::GlobalBarrierManager<S>::collect_barrier::{{closure}}
             at ./risingwave/src/meta/src/barrier/mod.rs:762:32
   7: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
   8: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
   9: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  10: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  11: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  12: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  13: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  14: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  15: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  16: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  17: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  18: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  19: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  20: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:394:9
  21: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:487:21
  22: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
  23: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
  24: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  25: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  26: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  27: tokio::macros::scoped_tls::ScopedKey<T>::set
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/macros/scoped_tls.rs:61:9
  28: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  29: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  30: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/task.rs:42:21
  31: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  32: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:223:17
  33: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/loom/std/unsafe_cell.rs:14:9
  34: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/core.rs:212:13
  35: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:476:19
  36: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  37: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  38: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  39: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  40: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:464:18
  41: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:198:27
  42: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/harness.rs:152:15
  43: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/raw.rs:200:18
  44: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/task/mod.rs:431:9
  45: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:159:9
  46: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:511:17
  47: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/blocking/pool.rs:469:13
  48: std::sys_common::backtrace::__rust_begin_short_backtrace
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys_common/backtrace.rs:121:18
  49: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:558:17
  50: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/panic/unwind_safe.rs:271:9
  51: std::panicking::try::do_call
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:483:40
  52: std::panicking::try
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panicking.rs:447:19
  53: std::panic::catch_unwind
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/panic.rs:140:14
  54: std::thread::Builder::spawn_unchecked_::{{closure}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/thread/mod.rs:557:30
  55: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/ops/function.rs:250:5
  56: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  57: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/alloc/src/boxed.rs:1988:9
  58: std::sys::unix::thread::Thread::new::thread_start
             at ./rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/std/src/sys/unix/thread.rs:108:17
  59: <unknown>
  60: __clone

2023-03-12T08:00:58.684896Z  INFO risingwave_meta::barrier::recovery: recovery start!
2023-03-12T08:00:59.163796Z  INFO risingwave_meta::barrier::recovery: recovery success

@lmatz lmatz added the type/bug Something isn't working label Mar 12, 2023
@BugenZhao
Copy link
Member

Seems still nothing helpful :(

Did any else error occur earlier in the compute node's log? Like Sink error or Remote sink error.

@tabVersion
Copy link
Contributor

the root cause may be on the connector node's log. It will be helpful to identify the problem.

@fuyufjh
Copy link
Member

fuyufjh commented Mar 13, 2023

the root cause may be on the connector node's log. It will be helpful to identify the problem.

I guess he had upload it in #8482 (comment)

image

@fuyufjh
Copy link
Member

fuyufjh commented Mar 13, 2023

Since the log didn't show the actual cause, is it possible to share the queries (like create table, create MVs, create sink, etc.) to allow us to reproduce it? @JuchangGit Thank you very much!

@liurenjie1024 liurenjie1024 changed the title CREATE SINK panic bug: CREATE SINK panic Mar 13, 2023
@tabVersion tabVersion mentioned this issue Mar 15, 2023
Merged
5 tasks
@tabVersion tabVersion modified the milestones: release-0.18, release-0.19 Mar 20, 2023
@tabVersion
Copy link
Contributor

@JuchangGit as we submitted a fix, could you please try on the latest main?

@JuchangGit
Copy link
Author

@JuchangGit as we submitted a fix, could you please try on the latest main?

抱歉，拖了这么久，今天刚抽出时间测试了。0.18版本已经没有问题了

@xxchan xxchan closed this as completed Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tabVersion tabVersion

Labels
type/bug Something isn't working
Projects
None yet
Milestone
release-0.19
Development

No branches or pull requests
6 participants
@lmatz @fuyufjh @tabVersion @BugenZhao @JuchangGit @xxchan