Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable mmap on 32-bit architectures #2000

Merged
merged 1 commit into from
May 19, 2023
Merged

Re-enable mmap on 32-bit architectures #2000

merged 1 commit into from
May 19, 2023

Conversation

adamreichold
Copy link
Contributor

memmap2 v0.3.0 introduced a regression when trying to map files larger than 4GB on 32-bit architectures which was subsequently fixed in v0.3.1.

This commit bumps locked version of the memmap2 dependency to the current v0.5.0 and reverts fdfc418 to re-enable mmap on 32-bit architectures as a different approach to fixing #1911.

This was tested to report matches from the end of a 5GB file using MinGW and Wine:

~/.wine/drive_c> wine rg.exe 3z5llj3n8b56dcpoj5aj4rlmq3bdpie wikidatawiki-20210901-pages-articles1.xml-p1p441397 --debug
DEBUG|grep_regex::literal|crates/regex/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(3z5llj3n8b56dcpoj5aj4rlmq3bdpie)], limit_size: 250, limit_class: 10 }
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: lz4: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: brotli: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: zstd: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: zstd: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: uncompress: could not find executable in PATH
DEBUG|grep_searcher::searcher::mmap|crates/searcher/src/searcher/mmap.rs:86: wikidatawiki-20210901-pages-articles1.xml-p1p441397: failed to open memory map: memory map length overflows usize
1727003:      <sha1>3z5llj3n8b56dcpoj5aj4rlmq3bdpie</sha1>

whereas still using memmap2 v0.3.0 but fdfc418 reverted would silently fail

~/.wine/drive_c> wine rg.exe 3z5llj3n8b56dcpoj5aj4rlmq3bdpie wikidatawiki-20210901-pages-articles1.xml-p1p441397 --debug
DEBUG|grep_regex::literal|crates/regex/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(3z5llj3n8b56dcpoj5aj4rlmq3bdpie)], limit_size: 250, limit_class: 10 }
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: lz4: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: brotli: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: zstd: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: zstd: could not find executable in PATH
DEBUG|grep_cli::decompress|crates/cli/src/decompress.rs:482: uncompress: could not find executable in PATH

memmap2 v0.3.0 introduced a regression when trying to map files larger than 4GB
on 32-bit architectures [1] which was subsequently fixed in v0.3.1 [2].

This commit bumps locked version of the memmap2 dependency to the current v0.5.0
and reverts fdfc418 to re-enable mmap on 32-bit
architectures as a different approach to fixing [3].

This was tested to report matches from the end of a 5GB file using MinGW and Wine.

[1] RazrFalcon/memmap2-rs@5e27122

[2] RazrFalcon/memmap2-rs@9aa838a

[3] #1911
@adamreichold
Copy link
Contributor Author

adamreichold commented May 19, 2023

I would be grateful for a short feedback whether this is wanted at all.

I think the root cause for the issues is highly likely to have been the bug in memmap2. Since that dependency has been bumped in the meantime, I don't think the workaround needs to be kept around.

@BurntSushi
Copy link
Owner

I think I buy it. Your research is enough to give me some confidence that this won't re-occur. If it does, we can re-apply the patch.

@BurntSushi BurntSushi merged commit 803c447 into BurntSushi:master May 19, 2023
@adamreichold adamreichold deleted the memmap-32-bits branch May 19, 2023 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants