Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removing FAISS from build #1340

Merged
merged 89 commits into from
Mar 18, 2023

Commits on Jan 27, 2023

  1. Replace faiss bfKnn

    Replace faiss bfKnn with code that leverages our pairwise_distance api
    and select_k api - by computing tiling over the inputs.
    
    This lets us remove faiss as a dependency
    benfred committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    57cfa20 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    984c641 View commit details
    Browse the repository at this point in the history
  3. fix merge

    benfred committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    805abc7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    3e21478 View commit details
    Browse the repository at this point in the history
  5. Fix bug with col_tiles < K

    benfred committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    74bd44f View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    c69054d View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2023

  1. Include metric_arg in bfknn

    benfred committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    1d9581b View commit details
    Browse the repository at this point in the history
  2. speedup compile times

    benfred committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    b4cf88c View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2023

  1. Configuration menu
    Copy the full SHA
    98ffb70 View commit details
    Browse the repository at this point in the history
  2. Suggestions from code review

    benfred committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    5442d31 View commit details
    Browse the repository at this point in the history
  3. fixes

    benfred committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    0f5d206 View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2023

  1. Configuration menu
    Copy the full SHA
    cb2b750 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2023

  1. Configuration menu
    Copy the full SHA
    e870eb3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cd84397 View commit details
    Browse the repository at this point in the history
  3. Use distance specializations

    benfred committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    8445aed View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2023

  1. Configuration menu
    Copy the full SHA
    e87633a View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2023

  1. Configuration menu
    Copy the full SHA
    52bf729 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d97ddb8 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2023

  1. Configuration menu
    Copy the full SHA
    5905b2d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d905266 View commit details
    Browse the repository at this point in the history
  3. use pw specializations in rbc

    benfred committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    8eaba84 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    fe728e9 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2023

  1. Configuration menu
    Copy the full SHA
    96e05e1 View commit details
    Browse the repository at this point in the history
  2. Revert "use pw specializations in rbc"

    This reverts commit 8eaba84.
    
    Change didn't seem to build in CI
    benfred committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    59060b2 View commit details
    Browse the repository at this point in the history
  3. Add tests for other metrics

    Also remove metrics processors - since is handled inside PW distance
    benfred committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    c734bac View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2023

  1. Fix parameter order

    benfred committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    c65e4bb View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2023

  1. Fix Lp distance

    benfred committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    3830e53 View commit details
    Browse the repository at this point in the history
  2. Revert "use matrix::select_k in bfknn call"

    This reverts commit fe728e9.
    
    This is causing incorrect results, just use the faiss select_k call
    instead
    benfred committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    3f0b9a7 View commit details
    Browse the repository at this point in the history
  3. re-enable failing tests

    benfred committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    3900570 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8e71915 View commit details
    Browse the repository at this point in the history
  5. Test JensenShannon distance

    benfred committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    f806bf6 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2023

  1. support k up to 2048 in faiss select

    The faiss repo allows k values up to 2048, but we were limiting
    to 512 or 1024 instead. This seems to be because of compiler
    errors such as
    
    ```
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb1ELi2048ELi8ELi64EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb0ELi2048ELi8ELi64EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb1ELi1024ELi8ELi128EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb0ELi1024ELi8ELi128EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ```
    
    when compiling with larger k values.
    
    However, these only affect double precision with 64bit indices - and float32
    works up to k=2048 (even with 64 bit indices).
    benfred committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    3315dca View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1b6eda2 View commit details
    Browse the repository at this point in the history
  3. cmake format

    benfred committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    a83bef3 View commit details
    Browse the repository at this point in the history
  4. support k up to 2048 in faiss select

    The faiss repo allows k values up to 2048, but we were limiting
    to 512 or 1024 instead. This seems to be because of compiler
    errors such as
    
    ```
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb1ELi2048ELi8ELi64EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb0ELi2048ELi8ELi64EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb1ELi1024ELi8ELi128EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ptxas error   : Entry function '_ZN4raft7spatial3knn6detail15select_k_kernelIdmLb0ELi1024ELi8ELi128EEEvPKT_PKT0_mmPS4_PS7_S4_S7_i' uses too much shared data (0x10000 bytes, 0xc000 max)
    ```
    
    when compiling with larger k values.
    
    However, these only affect double precision with 64bit indices - and float32
    works up to k=2048 (even with 64 bit indices).
    benfred committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    3b811a1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    9a19456 View commit details
    Browse the repository at this point in the history
  6. style

    benfred committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    c39dc65 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c60e17f View commit details
    Browse the repository at this point in the history
  8. code review suggestions

    benfred committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    2752294 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    84f7a42 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2023

  1. Configuration menu
    Copy the full SHA
    901b898 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2023

  1. Configuration menu
    Copy the full SHA
    1548a78 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3fdc712 View commit details
    Browse the repository at this point in the history
  3. temporarily re-add faiss build targets

    This is needed by the ANN benchmarks code, as well as cuml
    benfred committed Feb 21, 2023
    Configuration menu
    Copy the full SHA
    31c9cf2 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f7fd6a7 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2023

  1. re-add faiss_mr

    benfred committed Feb 22, 2023
    Configuration menu
    Copy the full SHA
    37d66d2 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2023

  1. explicitly include faiss_mr

    benfred committed Feb 23, 2023
    Configuration menu
    Copy the full SHA
    a61c92f View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2023

  1. Configuration menu
    Copy the full SHA
    dbd31b2 View commit details
    Browse the repository at this point in the history
  2. fix faiss queryempty test

    benfred committed Feb 24, 2023
    Configuration menu
    Copy the full SHA
    fddecc3 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2023

  1. exclude LP from fused

    benfred committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    4687144 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bd3ff51 View commit details
    Browse the repository at this point in the history
  3. use metric processor for cosine/correlation

    cuml expects cosine distance to be (1-Cosine(a,b)), where the
    pairwise_distances api was returning just Cosine(a,b). Revert
    benfred committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    06c8674 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2023

  1. exclude cosine

    benfred committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    b44d15c View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2023

  1. Configuration menu
    Copy the full SHA
    5a582cd View commit details
    Browse the repository at this point in the history
  2. avoid l2expanded distance

    I'm seeing a bunch of test failures in cuml CI - and at least some of them
    fail with L2Expanded distance and work with L2Unexpanded distance.
    benfred committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    eb0271a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    616455c View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2023

  1. Expanded L2 Changes

    Only calculate norms once per input (rather than per-tile) when
    calculating L2Expanded distance
    benfred committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    4c41c63 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2023

  1. Configuration menu
    Copy the full SHA
    cdf1962 View commit details
    Browse the repository at this point in the history
  2. warp divergence

    benfred committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    6a1e2d8 View commit details
    Browse the repository at this point in the history
  3. clamp to 0

    benfred committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    1e2817c View commit details
    Browse the repository at this point in the history
  4. threshold

    benfred committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    4b56fac View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2023

  1. Configuration menu
    Copy the full SHA
    4b41e2c View commit details
    Browse the repository at this point in the history
  2. fix

    benfred committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    455c952 View commit details
    Browse the repository at this point in the history
  3. Fix stream handling on col-major inputs

    For col-major inputs, we transpose the inputs since we only handle row-major
    in the tiled_brute_force_knn call.  However, this was happening on the stream
    from stream pool when given multiple partitions - causing dask errors later on.
    
    Fix.
    benfred committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    df46b65 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    97a3c01 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Configuration menu
    Copy the full SHA
    105bc96 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2828c3b View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2023

  1. Configuration menu
    Copy the full SHA
    6e45267 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2023

  1. fix build for missing symbols

    benfred committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    28ebeef View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9917324 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. code review feedback

    benfred committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    65d7725 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e41ff88 View commit details
    Browse the repository at this point in the history
  3. build fix

    benfred committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    5eb7d22 View commit details
    Browse the repository at this point in the history
  4. fix

    benfred committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    e36d089 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    50e366f View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e8f9c55 View commit details
    Browse the repository at this point in the history
  7. Removing faiss from build

    cjnolet committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    24004e3 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    6b01a17 View commit details
    Browse the repository at this point in the history
  9. Removing faissmr

    cjnolet committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    4d49ffd View commit details
    Browse the repository at this point in the history
  10. Removing faiss from docs

    cjnolet committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    5ce9230 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    9f211a0 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    be4ca13 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    9cbac3c View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2023

  1. Configuration menu
    Copy the full SHA
    0667526 View commit details
    Browse the repository at this point in the history
  2. Use stream pool

    benfred committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    9593ae1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    76d2b19 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    01fb23b View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2023

  1. Configuration menu
    Copy the full SHA
    d5debc7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c635073 View commit details
    Browse the repository at this point in the history