Predict 1.9-4.2x faster

[kpu]

I made prediction 1.9x to 4.2x faster than before.

Motivation

I want to use https://tinyurl.com/nllblid218e and similarly parametrized models to run language classification on petabytes of web data.

Methodology

The costliest operation is summing the rows for each model input. I've optimized this in three ways:

1. addRowToVector was a virtual function call for each row. I've replaced this with one virtual function call per prediction by adding averageRowsToVector to Matrix calls.
2. Vector and DenseMatrix were not 64-byte aligned so the CPU was doing a lot of unaligned memory access. I've brought in my own vector replacement that does 64-byte alignment.
3. Write the averageRowsToVector in intrinsics for common vector sizes. This works on SSE, AVX, and AVX512F.

See the commit history for a breakdown of speed improvement from each change.

Experiments

Test set docs1000.txt.gz which is a bunch of random documents https://data.statmt.org/heafield/classified-fasttext/
CPU: AMD Ryzen 9 7950X 16-Core

Model https://tinyurl.com/nllblid218e with 256-dimensional vectors
Before
real 0m8.757s
user 0m8.434s
sys 0m0.327s

After
real 0m2.046s
user 0m1.717s
sys 0m0.334s

Model https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin with 16-dimensional vectors
Before
real 0m0.926s
user 0m0.889s
sys 0m0.037s

After
real 0m0.477s
user 0m0.436s
sys 0m0.040s

[facebook-github-bot]

Hi @kpu!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[kpu]

cc @Celebio

[plo-]

Any update @kpu?

[facebook-github-bot]

@kpuatfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[ZJaume]

FasterText is failing in the Python module build because Setuptools does not set the c++17 flag, giving this kind of errors:

src/dictionary.h:75:28: error: ‘string_view’ in namespace ‘std’ does not name a type
   75 |   int32_t getId(const std::string_view) const;
      |                            ^~~~~~~~~~~
src/dictionary.h:75:23: note: ‘std::string_view’ is only available from C++17 onwards
   75 |   int32_t getId(const std::string_view) const;
      |                       ^~~
src/dictionary.h:76:28: error: ‘string_view’ in namespace ‘std’ does not name a type
   76 |   int32_t getId(const std::string_view, uint32_t h) const;

Changing the flag in setup.py fixes it:

diff --git a/setup.py b/setup.py
index 50c166b..8aff0b5 100644
--- a/setup.py
+++ b/setup.py
@@ -98,15 +98,14 @@ def has_flag(compiler, flags):


 def cpp_flag(compiler):
-    """Return the -std=c++[11/14] compiler flag.
-    The c++14 is preferred over c++11 (when it is available).
+    """Return the -std=c++17 compiler flag.
     """
-    standards = ['-std=c++11']
+    standards = ['-std=c++17']
     for standard in standards:
         if has_flag(compiler, [standard]):
             return standard
     raise RuntimeError(
-        'Unsupported compiler -- at least C++11 support '
+        'Unsupported compiler -- at least C++17 support '
         'is needed!'
     )

[facebook-github-bot]

@kpuatfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@kpuatfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@kpuatfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@kpuatfb merged this pull request in b733943.