Unrecoverable issue: "Last written record was unable to be deserialized. Corruption likely."

[NicolasFloquet]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

This issue most likely happened after an un-graceful shutdown of the kubernetes node vector was running on.
Now the pod keeps crashing will following errors:
023-02-21T10:51:45.658120Z ERROR vector_buffers::variants::disk_v2::writer: Last written record was unable to be deserialized. Corruption likely. reason="invalid data : check failed for struct member payload: subtree pointer out of bounds: ptr 0x7ff00a60da58 not in range 0x7ff00374f000..0x7ff0059ffa00"

2023-02-21T10:51:57.324468Z ERROR vector::topology: Configuration error. error=Sink "eventhub_sink": error occurred when building buffer: failed to build individual stage 0: failed to seek to position where reader left off: failed to deserialize encoded record from buffer: invalid data: check failed for struct member payload: subtree pointer out of bounds: ptr 0x7ff00a6340d8 not in range 0x7ff00a64f280..0x7ff00a650880

Configuration

sources:
    vector_source:
      type: vector
      address: 0.0.0.0:9000
      version: "2"
  sinks:
    eventhub_sink:
      type: kafka
      buffer:
        type: "disk"
        max_size: 536870912
      inputs:
        - vector_source
      encoding: 
        codec: json
      bootstrap_servers: "${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093"
      topic: topic-name
      sasl:
        enabled: true
        mechanism: "PLAIN"
        username: "$$ConnectionString"
        password: "${EVENTHUB_CONNECTION_STRING}"
      healthcheck: true
      compression: none
      message_timeout_ms: 0
      librdkafka_options:
        "security.protocol": sasl_ssl
        "socket.keepalive.enable": "true"
        "metadata.max.age.ms": "180000"
        "request.timeout.ms": "30000"
        "partitioner": "consistent_random"

Version

0.27.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[StephenWakely]

Related #16463

[jszwedko]

Blocked by #16754 to allow us to reproduce this issue in order to resolve it.