Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Java/C++] Archive variable length recordings. #1069

Merged

Conversation

vyazelenko
Copy link
Contributor

This PR changes Archive recordings from being fixed length (1KB) to variable-length.

Summary:

  • Recording descriptors are variable-length.
  • No limitation on the length of the strippedChannel, originalChannel and sourceIdentity Strings.
  • Ability to purge recordings (io.aeron.archive.client.AeronArchive#purgeRecording), i.e. mark recording as INVALID and delete the corresponding segment files.
  • max-entries commands in ArchiveTool have been deprecated in favor of capacity, i.e. Catalog no longer has a max entries limit but has a capacity expressed in bytes. Catalog is auto-growing it's capacity when needed but an explicit command can be used as well.
  • New command ArchiveTool#compact to compact the Catalog file, i.e. remove all recordings in state INVALID and delete the corresponding segment files.
  • Add checksum to the RecordingDescriptorHeader. Now if the io.aeron.archive.Archive.Context#recordChecksum() is set the checksum of the entire RecordingDescriptor will be taken and stored in the header.
  • ArchiveTool#verify - validate RecordingDescriptor checksums.
  • ArchiveTool#checksum - compute and persist RecordingDescriptor checksums.
  • Add support for jumbo RecordingDescriptors, i.e. allow RecordingDescriptors bigger than the tryClaim size by using Publication#offer API.
  • Migration task to migrate from version 2 to version 3 of the Archive.

@vyazelenko vyazelenko force-pushed the archive-variable-length-recordings branch 2 times, most recently from 971579a to 09ff33e Compare October 5, 2020 16:06
@vyazelenko vyazelenko force-pushed the archive-variable-length-recordings branch 2 times, most recently from 71ac442 to ce600b7 Compare October 26, 2020 11:33
@vyazelenko vyazelenko force-pushed the archive-variable-length-recordings branch from ce600b7 to 4df6384 Compare October 28, 2020 08:36
…ding (i.e. Catalog#maxDescriptorStringsCombinedLength).
…N_RECORDING` when recording not found or `minRecordingId` is out of range.
…scriptor) + compute alignment/nextRecordingId for old catalog files.
…y#started to support 'jumbo' channel URIs.
Address `java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open` which is thrown on Windows.
@vyazelenko vyazelenko force-pushed the archive-variable-length-recordings branch from 4df6384 to 3d1f621 Compare November 12, 2020 09:54
@mjpt777 mjpt777 merged commit 0a57f05 into real-logic:master Nov 13, 2020
@vyazelenko vyazelenko deleted the archive-variable-length-recordings branch November 27, 2020 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants