Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix incorrect double-dequeue after restarting #12

Merged
merged 1 commit into from
Nov 6, 2023

Conversation

izquierdo
Copy link
Contributor

Problem

After calling Close() on a queue and then reopening the same data directory, it's possible for entries that were already dequeued to show up again in the queue.

Expected behavior

Entries that were dequeued should not show up again in a subsequent Dequeue() call.

Details

  1. Currently, loadFileInfos() reads all files in the data directory that have the seg_ prefix, including offset files.
  2. loadFileInfos() sorts those files by modification timestamp.
  3. When the queue is closed, an 8-byte "segment ending" marker is written to the data file, but the offset file is not touched further. This means that the modification timestamp for the data file ends up being later than that of the offset file.
  4. After the queue is reopened, since the offset file has an earlier modification timestamp it's placed earlier than the data file in loadFileInfo()'s return.
  5. Thus, when attempting to dequeue, the offset file is read first but reading fails since the offset file doesn't follow the same format as the data file (depending on luck, it might be possible for the offset file to be interpreted as if it was a data file).
  6. Since reading the offset file fails, the offset file is deleted as part of cleanup of the segment structure. The data file remains behind.
  7. dequeue() then tries the next file in the list, which is the data file. By this point, the offset file has already been deleted, so dequeue() incorrectly reads the data file from the start instead of seeking to the correct offset.

@linxGnu
Copy link
Owner

linxGnu commented Nov 6, 2023

That's awesome finding. Thank you so much @izquierdo

@linxGnu linxGnu merged commit 963d19a into linxGnu:main Nov 6, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants