Limiting total rows copied in COPY TABLE FROM statement

[v0y4g3r]

What problem does the new feature solve?

As pointed out by this article, a 42 KiB parquet file may contain hundreds of trillions of values, just like what zip bomb does. If GreptimeDB does not limit the total rows copied in single COPY TABLE FROM statement, these parquet bombs may immediately overload the backend storage.

What does the feature do?

Limit total row imported by single COPY TABLE FROM statement.

The execution of copy statements will build a record batch stream on file and insert the batches yielded to mito engine.

greptimedb/src/operator/src/statement/copy_table_from.rs

Lines 402 to 437 in 3acd5bf

	while let Some(r) = stream.next().await {
	let record_batch = r.context(error::ReadDfRecordBatchSnafu)?;
	let vectors =
	Helper::try_into_vectors(record_batch.columns()).context(IntoVectorsSnafu)?;
	pending_mem_size += vectors.iter().map(|v| v.memory_size()).sum::<usize>();
	let columns_values = fields
	.iter()
	.cloned()
	.zip(vectors)
	.collect::<HashMap<_, _>>();
	pending.push(self.inserter.handle_table_insert(
	InsertRequest {
	catalog_name: req.catalog_name.to_string(),
	schema_name: req.schema_name.to_string(),
	table_name: req.table_name.to_string(),
	columns_values,
	},
	query_ctx.clone(),
	));
	if pending_mem_size as u64 >= pending_mem_threshold {
	let (rows, cost) = batch_insert(&mut pending, &mut pending_mem_size).await?;
	rows_inserted += rows;
	insert_cost += cost;
	}
	}
	if !pending.is_empty() {
	let (rows, cost) = batch_insert(&mut pending, &mut pending_mem_size).await?;
	rows_inserted += rows;
	insert_cost += cost;
	}
	}

We can add a config option to limit the total rows read from the record batch stream and terminate the execution once it exceeds threshold.

[irenjj]

I'd like to take it, Could you please assign it to me?

[v0y4g3r]

I'd like to take it, Could you please assign it to me?

Sure. Are you going to add a new runtime option to limit the rows?

[irenjj]

Sure. Are you going to add a new runtime option to limit the rows?

Sure, should we add a configuration option in the OptionMap in CopyTableArgument to limit it?

[v0y4g3r]

Sure. Are you going to add a new runtime option to limit the rows?

Sure, should we add a configuration option in the OptionMap in CopyTableArgument to limit it?

Maybe we can set a default limit, for example 1000 rows, unless user explicitly specified the limit in COPY statement.