Support non-consecutive offsets as they might be present in compacted topics

[Astrac]

No description provided.

[amrutha-shanbhag]

@Astrac thanks for submitting the PR. We have added it into our internal work backlog. We should be able to address it latest by next month.

[h3nd24]

Hi @Astrac sorry it took us a while. For the change, I think the original condition of consecutive offset have some values in validating non-compacted topics. As such, I would propose to add a config property (e.g., in AwsStorageConnectorCommonConfig) instead to indicate whether the topic is compacted or not. If the topic is compacted, then we skip the check of consecutive offset. What do you think? We can even make your proposed behaviour to be the default, and make the consecutive validation as an opt in

[h3nd24]

make it a configurable whether we want consecutive offset or not

[debaditya-netapp]

Hi @Astrac, Is this still being worked upon?