iceberg: fix spec inconsistency in manifest list files_count

[andrwng]

The schema we are using was pulled some time ago and appears to be outdated. The Apache Iceberg Java implementation has since renamed added_data_files_count and friends to added_files_count, to match the documented spec.

This meant that after updating the table with an external non-Redpanda writer, Redpanda wouldn't be able to download the current manifest list when appending and get stuck, complaining about an EOF (presumably the Avro C++ library throws this when there is an unknown field).

I suspect that this may have been the cause of an EOF seen when trying to read a manifest list with BigQuery:

Error while reading data, error message: The Apache Avro failed to read data with the following error: EOF reached File: [...]/metadata/snap-[...]-0.avro

The old names are added as an alias to ensure Redpanda can still download Iceberg manifest lists from 24.3. This PR also adds an upgrade test to validate that this alias is actually required, rather than simply renaming the field. Without the alias, the upgrade test fails.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

Bug Fixes

• Fixes a bug in Redpanda's Iceberg manifest list Avro definition that previously resulted in an end-of-file (EOF) error when reading manifest list Avro files written by other engines. This could previously block Redpanda appending Iceberg data, and could also prevent certain query engines from successfully reading Iceberg data written by Redpanda.

[vbotbuildovich]

Retry command for Build#59894

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/partition_movement_test.py::PartitionMovementTest.test_cross_core_movements@{"cloud_storage_type":1}
tests/rptest/tests/cloud_retention_test.py::CloudRetentionTest.test_cloud_retention@{"cloud_storage_type":2,"max_consume_rate_mb":20}

[vbotbuildovich]

CI test results

test results on build#59894

test_id	test_kind	job_url	test_status	passed
rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/59894#0193d7dc-498b-46b5-9e78-c05a95af3147	FAIL	0/6
rptest.tests.datalake.partition_movement_test.PartitionMovementTest.test_cross_core_movements.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/59894#0193d7dc-4989-42dd-ad69-30935788b832	FAIL	0/6

[ztlpn]

Hmm I wonder if we should backport this. Technically this is a breaking change that will prevent downgrades to earlier 24.3 versions. Or is backward compat also working thanks to aliases?

[andrwng]

Hmm I wonder if we should backport this. Technically this is a breaking change that will prevent downgrades to earlier 24.3 versions. Or is backward compat also working thanks to aliases?

Yeah good thought. You're right, this will prevent Iceberg from working following a rollback, even with the alias, because 24.3.prev won't be able to read the manifest lists with the new Avro schema.

That said, I lean towards it being justifiable -- it's a correctness fix, especially for a beta feature.