[Bug]: BigQueryIO direct read not reading all rows when set --setEnableBundling=true

[Abacn]

What happened?

The feature is introduced in #25392. When --setEnableBundling=true pipeline option is set, it turns out that BigQueryIO only reads a small fraction of row for large table. Reproduced reading tpcds_1T.web_sales table.

Number of rows: 720,000,376
--setEnableBundling=true: 44,550,489 rows read
--setEnableBundling=false: 720,000,376 rows read

Reading from tpcds_1G.web_sales table, the issue is not triggered, as 18,000 rows are read.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Affecting >=v2.46.0

demo that reproduces this issue: https://github.com/Abacn/beam-demo/tree/bigqueryiotestbranch

[kennknowles]

This seems like a pretty severe issue. Any progress?

[Abacn]

The feature is introduced in Beam v2.46.0 and is activated only when this currently undocumented pipeline option is set. No production user is using it. @vachan-shetty is working on fix. Feel free to assign to yourself.

[kennknowles]

Is this related to #26521 or no?

[Abacn]

@kennknowles thanks, will test if #26503 fixed the issue.

Update, no, that issue is for write, this is for read