api: add memmem implementation, initially from bstr

This commit primarily adds vectorized substring search routines in
a new memmem sub-module. They were originally taken from bstr, but
heavily modified to incorporate a variant of the "generic SIMD"
algorithm[1]. The main highlights:

* We guarantee `O(m + n)` time complexity and constant space
  complexity.
* Two-Way is the primary implementation that can handle all cases.
* Vectorized variants handle a number of common cases.
* Vectorized code uses a heuristic informed by a frequency background
  distribution of bytes, originally devised inside the regex crate.
  This makes it more likely that searching will spend more time in the
  fast vector loops.

While adding memmem to this crate is perhaps a bit of a scope increase,
I think it fits well. It also puts a core primitive, substring
search, very low in the dependency DAG and therefore making it widely
available. For example, it is intended to use these new routines in the
regex, aho-corasick and bstr crates.

This commit does a number of other things, mainly as a result of
convenience. It drastically improves test coverage for substring search
(as compared to what bstr had), completely overhauls the benchmark suite
to make it more comprehensive and adds `cargo fuzz` support for all API
items in the crate.

[1] - http://0x80.pl/articles/simd-strfind.html#algorithm-1-generic-simd

.github/workflows/ci.yml

@@ -15,6 +15,8 @@ jobs:
       CARGO: cargo
       # When CARGO is set to CROSS, TARGET is set to `--target matrix.target`.
       TARGET:
+      # Make quickcheck run more tests for hopefully better coverage.
+      QUICKCHECK_TESTS: 100000
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:

Cargo.toml

@@ -37,8 +37,12 @@ libc = { version = "0.2.18", default-features = false, optional = true }
 [dev-dependencies]
 quickcheck = { version = "1.0.3", default-features = false }
 
+[profile.release]
+debug = true
+
 [profile.bench]
 debug = true
 
 [profile.test]
 opt-level = 3
+debug = true

README.md

@@ -1,6 +1,6 @@
 memchr
 ======
-The `memchr` crate provides heavily optimized routines for searching bytes.
+This library provides heavily optimized routines for string search primitives.
 
 [![Build status](https://github.com/BurntSushi/rust-memchr/workflows/ci/badge.svg)](https://github.com/BurntSushi/rust-memchr/actions)
 [![](https://meritbadge.herokuapp.com/memchr)](https://crates.io/crates/memchr)
@@ -15,23 +15,15 @@ Dual-licensed under MIT or the [UNLICENSE](https://unlicense.org/).
 
 ### Overview
 
-The `memchr` function is traditionally provided by libc, but its
-performance can vary significantly depending on the specific
-implementation of libc that is used. They can range from manually tuned
-Assembly implementations (like that found in GNU's libc) all the way to
-non-vectorized C implementations (like that found in MUSL).
+* The top-level module provides routines for searching for 1, 2 or 3 bytes
+  in the forward or reverse direction. When searching for more than one byte,
+  positions are considered a match if the byte at that position matches any
+  of the bytes.
+* The `memmem` sub-module provides forward and reverse substring search
+  routines.
 
-To smooth out the differences between implementations of libc, at least
-on `x86_64` for Rust 1.27+, this crate provides its own implementation of
-`memchr` that should perform competitively with the one found in GNU's libc.
-The implementation is in pure Rust and has no dependency on a C compiler or an
-Assembler.
-
-Additionally, GNU libc also provides an extension, `memrchr`. This crate
-provides its own implementation of `memrchr` as well, on top of `memchr2`,
-`memchr3`, `memrchr2` and `memrchr3`. The difference between `memchr` and
-`memchr2` is that `memchr2` permits finding all occurrences of two bytes
-instead of one. Similarly for `memchr3`.
+In all such cases, routines operate on `&[u8]` without regard to encoding. This
+is exactly what you want when searching either UTF-8 or arbitrary bytes.
 
 ### Compiling without the standard library
 
@@ -58,11 +50,11 @@ using `memchr` from libc is desirable and a vectorized routine is not otherwise
 available in this crate, then enabling the `libc` feature will use libc's
 version of `memchr`.
 
-The rest of the functions in this crate, e.g., `memchr2` or `memrchr3`, are not
-a standard part of libc, so they will always use the implementations in this
-crate. One exception to this is `memrchr`, which is an extension commonly found
-on Linux. On Linux, `memrchr` is used in precisely the same scenario as
-`memchr`, as described above.
+The rest of the functions in this crate, e.g., `memchr2` or `memrchr3` and the
+substring search routines, will always use the implementations in this crate.
+One exception to this is `memrchr`, which is an extension in `libc` found on
+Linux. On Linux, `memrchr` is used in precisely the same scenario as `memchr`,
+as described above.
 
 
 ### Minimum Rust version policy
@@ -77,3 +69,20 @@ version of Rust.
 
 In general, this crate will be conservative with respect to the minimum
 supported version of Rust.
+
+
+### Testing strategy
+
+Given the complexity of the code in this crate, along with the pervasive use
+of `unsafe`, this crate has an extensive testing strategy. It combines multiple
+approaches:
+
+* Hand-written tests.
+* Exhaustive-style testing meant to exercise all possible branching and offset
+  calculations.
+* Property based testing through [`quickcheck`](https://github.com/BurntSushi/quickcheck).
+* Fuzz testing through [`cargo fuzz`](https://github.com/rust-fuzz/cargo-fuzz).
+* A huge suite of benchmarks that are also run as tests. Benchmarks always
+  confirm that the expected result occurs.
+
+Improvements to the testing infrastructue are very welcome.

bench/Cargo.toml

@@ -18,6 +18,10 @@ harness = false
 path = "src/bench.rs"
 
 [dependencies]
+bstr = "0.2.15"
 criterion = "0.3.3"
 memchr = { version = "*", path = ".." }
 libc = "0.2.81"
+regex = "1.4.5"
+sliceslice = "0.2.1"
+twoway = "0.2.1"

bench/data/opensubtitles/README.md

@@ -0,0 +1,12 @@
+These were downloaded and derived from the Open Subtitles data set:
+https://opus.nlpl.eu/OpenSubtitles-v2018.php
+
+The specific way in which they were modified has been lost to time, but it's
+likely they were just a simple truncation based on target file sizes for
+various benchmarks.
+
+The main reason why we have them is that it gives us a way to test similar
+inputs on non-ASCII text. Normally this wouldn't matter for a substring search
+implementation, but because of the heuristics used to pick a priori determined
+"rare bytes" to base a prefilter on, it's possible for this heuristic to do
+more poorly on non-ASCII text than one might expect.