Skip to content

Commit

Permalink
Preserve records batch order when SchemaCastScanExec is involved (#20)
Browse files Browse the repository at this point in the history
  • Loading branch information
sgrebnov authored and phillipleblanc committed Oct 12, 2024
1 parent e1084a9 commit 8f17eb8
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion datafusion-federation/src/schema_cast/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ use datafusion::error::{DataFusionError, Result};
use datafusion::execution::{SendableRecordBatchStream, TaskContext};
use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
use datafusion::physical_plan::{
DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, PlanProperties,
DisplayAs, DisplayFormatType, Distribution, ExecutionPlan, ExecutionPlanProperties,
PlanProperties,
};
use futures::StreamExt;
use std::any::Any;
Expand Down Expand Up @@ -69,6 +70,12 @@ impl ExecutionPlan for SchemaCastScanExec {
vec![&self.input]
}

/// Prevents the introduction of additional `RepartitionExec` and processing input in parallel.
/// This guarantees that the input is processed as a single stream, preserving the order of the data.
fn benefits_from_input_partitioning(&self) -> Vec<bool> {
vec![false]
}

fn with_new_children(
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
Expand Down

0 comments on commit 8f17eb8

Please sign in to comment.