Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] Release version 2.14.0 #1576

Closed
8 of 23 tasks
opensearch-trigger-bot bot opened this issue Mar 23, 2024 · 8 comments
Closed
8 of 23 tasks

[RELEASE] Release version 2.14.0 #1576

opensearch-trigger-bot bot opened this issue Mar 23, 2024 · 8 comments
Assignees
@jmazanec15
Labels
v2.14.0

Comments

@opensearch-trigger-bot
Copy link
Contributor

opensearch-trigger-bot bot commented Mar 23, 2024 •
edited by rishabh6788
Loading

This is a component issue for 2.14.0.
Coming from opensearch-project/opensearch-build#4562. Please follow the following checklist.
Please refer to the DATES in that post.

How to use this issue

This Component Release Issue

This issue captures the state of the OpenSearch release, on component/plugin level; its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help.
Any release related work can be linked to this issue or added as comments to create visiblity into the release status.

Release Steps

There are several steps to the release process; these steps are completed as the whole component release and components that are behind present risk to the release. The component owner resolves the tasks in this issue and communicate with the overall release owner to make sure each component are moving along as expected.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release. The most current set of dates is on the overall release issue linked at the top of this issue.

The Overall Release Issue

Linked at the top of this issue, the overall release issue captures the state of the entire OpenSearch release including references to this issue, the release owner which is the assignee is responsible for communicating the release status broadly. Please contact them or @mention them on that issue for help.

What should I do if my plugin isn't making any changes?

If including changes in this release, increment the version on 2.x branch to 2.14.0 for Min/Core, and 2.14.0.0 for components. Otherwise, keep the version number unchanged for both.

Preparation

  • Assign this issue to a release owner.
  • Finalize scope and feature set and update the Public Roadmap.
  • All the tasks in this issue have been reviewed by the release owner.
  • Create, update, triage and label all features and issues targeted for this release with v2.14.0.
  • Finalize the code and create the the release branch 2.14 from the 2.x branch.

CI/CD

  • All code changes for 2.14.0 are complete.
  • Ensure working and passing CI.
  • Check that this repo is included in the distribution manifest.

Pre-Release

  • Increment the version on the parent branch to the next development iteration.
  • Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
  • Confirm that all changes for 2.14.0 have been merged.
  • Add this repo to the manifest for the next developer iteration.

Release Testing

  • Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
  • Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
  • Sanity Testing: Sanity testing and fixing of critical issues found.
  • File issues for all intermittent test failures.

Release

  • Complete documentation.
  • Verify all issued labeled for this release are closed or labelled for the next release.
  • Verify the release date mentioned in release notes is correct and matches actual release date.

Post Release

  • Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
  • Add this repo to the manifest of the next eventual security patch version.
  • Suggest improvements to this template.
  • Conduct a retrospective, and publish its results.
@jmazanec15 jmazanec15 self-assigned this Apr 4, 2024
@rishabh6788 rishabh6788 mentioned this issue Apr 30, 2024
Closed
72 tasks
@jmazanec15
Copy link
Member

faiss-hnsw-nested.json

{
  "metadata": {
    "test_name": "Faiss HNSW Nested Field Test",
    "test_id": "Faiss HNSW Nested Field Test",
    "date": "05/09/2024 19:51:07",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5117800448 (used) / 26980814848 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 414255.62760462036,
    "delete_index_took_total": 379.392886980592,
    "create_index_took_total": 383.74740733221796,
    "ingest_nested_field_took_total": 285104.4919160001,
    "refresh_index_store_kb_total": 2813655.7548828125,
    "refresh_index_took_total": 6396.543015978144,
    "force_merge_took_total": 68427.99320500733,
    "warmup_operation_took_total": 500.79250665536773,
    "query_nested_field_took_total": 53062.666666666664,
    "query_nested_field_took_p50": 5.0,
    "query_nested_field_took_p90": 6.0,
    "query_nested_field_took_p99": 8.0,
    "query_nested_field_took_p99.9": 15.0,
    "query_nested_field_took_p100": 739.6666666666666,
    "query_nested_field_client_time_total": 720215.6666666666,
    "query_nested_field_client_time_p50": 71.33333333333333,
    "query_nested_field_client_time_p90": 72.33333333333333,
    "query_nested_field_client_time_p99": 75.33333333333333,
    "query_nested_field_client_time_p99.9": 194.0,
    "query_nested_field_client_time_p100": 917.0,
    "query_nested_field_memory_kb_total": 1252217.3333333333,
    "query_nested_field_recall@K_total": 0.9992143333333333,
    "query_nested_field_recall@1_total": 1.0
  }
}

faiss-hnswpq.json

{
  "metadata": {
    "test_name": "Faiss HNSW PQ Test",
    "test_id": "Faiss HNSW PQ Test",
    "date": "05/09/2024 21:30:12",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5141807104 (used) / 26956677120 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 302775.3934569387,
    "delete_model_took_total": 193.9618436542029,
    "delete_index_took_total": 341.35504266790423,
    "train_model_took_total": 16448.81149266924,
    "create_index_took_total": 400.4267803199279,
    "ingest_took_total": 192090.72619663007,
    "refresh_index_store_kb_total": 1944384.0810546875,
    "refresh_index_took_total": 4075.9496550114513,
    "force_merge_took_total": 49066.192499012686,
    "warmup_operation_took_total": 212.30328030651435,
    "query_took_total": 39945.666666666664,
    "query_took_p50": 4.0,
    "query_took_p90": 4.0,
    "query_took_p99": 7.0,
    "query_took_p99.9": 8.666666666666666,
    "query_took_p100": 822.6666666666666,
    "query_client_time_total": 684069.6666666666,
    "query_client_time_p50": 68.0,
    "query_client_time_p90": 69.0,
    "query_client_time_p99": 71.0,
    "query_client_time_p99.9": 167.33333333333334,
    "query_client_time_p100": 911.6666666666666,
    "query_memory_kb_total": 323200.6666666667,
    "query_recall@K_total": 0.6291483333333333,
    "query_recall@1_total": 0.9947
  }
}

faiss-hnsw-relaxed-filter.json

{
  "metadata": {
    "test_name": "Faiss HNSW Relaxed Filter Test",
    "test_id": "Faiss HNSW Relaxed Filter Test",
    "date": "05/09/2024 18:05:33",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5129351168 (used) / 26969133056 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 333481.4301929508,
    "delete_index_took_total": 350.8955120147827,
    "create_index_took_total": 418.04611797366914,
    "ingest_multi_field_took_total": 193444.4566889821,
    "refresh_index_store_kb_total": 2888661.5618489585,
    "refresh_index_took_total": 5064.482664650616,
    "force_merge_took_total": 76043.48953898686,
    "warmup_operation_took_total": 534.3930036760867,
    "query_with_filter_took_total": 57625.666666666664,
    "query_with_filter_took_p50": 5.666666666666667,
    "query_with_filter_took_p90": 6.333333333333333,
    "query_with_filter_took_p99": 8.666666666666666,
    "query_with_filter_took_p99.9": 23.333333333333332,
    "query_with_filter_took_p100": 743.3333333333334,
    "query_with_filter_client_time_total": 755810.0,
    "query_with_filter_client_time_p50": 75.0,
    "query_with_filter_client_time_p90": 76.33333333333333,
    "query_with_filter_client_time_p99": 79.66666666666667,
    "query_with_filter_client_time_p99.9": 229.66666666666666,
    "query_with_filter_client_time_p100": 841.3333333333334,
    "query_with_filter_memory_kb_total": 1270411.6666666667,
    "query_with_filter_recall@K_total": 0.9990963333333333,
    "query_with_filter_recall@1_total": 1.0
  }
}

faiss-hnsw-restrictive-filter.json

{
  "metadata": {
    "test_name": "Faiss HNSW Restrictive Filter Test",
    "test_id": "Faiss HNSW Restrictive Filter Test",
    "date": "05/09/2024 18:55:51",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5094961152 (used) / 27003654144 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 324721.17707864754,
    "delete_index_took_total": 369.8509643242384,
    "create_index_took_total": 384.83532399792847,
    "ingest_multi_field_took_total": 199000.6200222997,
    "refresh_index_store_kb_total": 3038274.7102864585,
    "refresh_index_took_total": 4648.569527353781,
    "force_merge_took_total": 66717.26112399483,
    "warmup_operation_took_total": 504.0401166770607,
    "query_with_filter_took_total": 53096.0,
    "query_with_filter_took_p50": 5.0,
    "query_with_filter_took_p90": 5.666666666666667,
    "query_with_filter_took_p99": 10.0,
    "query_with_filter_took_p99.9": 19.666666666666668,
    "query_with_filter_took_p100": 796.6666666666666,
    "query_with_filter_client_time_total": 709549.6666666666,
    "query_with_filter_client_time_p50": 70.66666666666667,
    "query_with_filter_client_time_p90": 71.66666666666667,
    "query_with_filter_client_time_p99": 75.66666666666667,
    "query_with_filter_client_time_p99.9": 195.0,
    "query_with_filter_client_time_p100": 888.0,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

faiss-hnsw.json

{
  "metadata": {
    "test_name": "Faiss HNSW Test",
    "test_id": "Faiss HNSW Test",
    "date": "05/09/2024 20:41:28",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5012664320 (used) / 27086082048 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 314180.4423973418,
    "delete_index_took_total": 379.4440556860839,
    "create_index_took_total": 369.39972332523513,
    "ingest_took_total": 192665.63925065566,
    "refresh_index_store_kb_total": 2885292.5693359375,
    "refresh_index_took_total": 5242.1784996598335,
    "force_merge_took_total": 73197.51777234099,
    "warmup_operation_took_total": 491.5964290072831,
    "query_took_total": 41834.666666666664,
    "query_took_p50": 4.0,
    "query_took_p90": 4.0,
    "query_took_p99": 7.0,
    "query_took_p99.9": 9.0,
    "query_took_p100": 763.0,
    "query_client_time_total": 709797.3333333334,
    "query_client_time_p50": 71.0,
    "query_client_time_p90": 71.66666666666667,
    "query_client_time_p99": 74.0,
    "query_client_time_p99.9": 143.0,
    "query_client_time_p100": 852.6666666666666,
    "query_memory_kb_total": 1225265.6666666667,
    "query_recall@K_total": 0.9991846666666667,
    "query_recall@1_total": 1.0
  }
}

faiss-ivf.json

{
  "metadata": {
    "test_name": "Faiss IVF",
    "test_id": "Faiss IVF",
    "date": "05/09/2024 23:53:55",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5030772736 (used) / 27068035072 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 238239.0097729677,
    "delete_model_took_total": 206.32987065861622,
    "delete_index_took_total": 369.61718898965046,
    "train_model_took_total": 1794.1008456788647,
    "create_index_took_total": 370.2277599950321,
    "ingest_took_total": 188423.3086600046,
    "refresh_index_store_kb_total": 2589861.1927083335,
    "refresh_index_took_total": 451.08469732804224,
    "force_merge_took_total": 5377.066924314325,
    "warmup_operation_took_total": 464.9404926652399,
    "query_took_total": 40782.333333333336,
    "query_took_p50": 4.0,
    "query_took_p90": 4.0,
    "query_took_p99": 7.0,
    "query_took_p99.9": 10.333333333333334,
    "query_took_p100": 458.6666666666667,
    "query_client_time_total": 700683.3333333334,
    "query_client_time_p50": 69.66666666666667,
    "query_client_time_p90": 70.66666666666667,
    "query_client_time_p99": 73.0,
    "query_client_time_p99.9": 139.66666666666666,
    "query_client_time_p100": 547.3333333333334,
    "query_memory_kb_total": 1020766.0,
    "query_recall@K_total": 0.9496856666666668,
    "query_recall@1_total": 1.0
  }
}

faiss-ivfpq.json

{
  "metadata": {
    "test_name": "Faiss IVF PQ Test",
    "test_id": "Faiss IVF PQ Test",
    "date": "05/10/2024 00:42:23",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5093175296 (used) / 27005374464 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 262696.1867353013,
    "delete_model_took_total": 201.9214856554754,
    "delete_index_took_total": 330.9080090063314,
    "train_model_took_total": 17409.788996000618,
    "create_index_took_total": 362.5047176804704,
    "ingest_took_total": 196674.76097630183,
    "refresh_index_store_kb_total": 1759828.7236328125,
    "refresh_index_took_total": 559.658513967103,
    "force_merge_took_total": 5987.8775850132415,
    "warmup_operation_took_total": 200.76645167622095,
    "query_took_total": 40968.0,
    "query_took_p50": 4.0,
    "query_took_p90": 4.0,
    "query_took_p99": 6.666666666666667,
    "query_took_p99.9": 8.666666666666666,
    "query_took_p100": 635.6666666666666,
    "query_client_time_total": 719070.3333333334,
    "query_client_time_p50": 72.0,
    "query_client_time_p90": 72.33333333333333,
    "query_client_time_p99": 75.0,
    "query_client_time_p99.9": 155.66666666666666,
    "query_client_time_p100": 726.6666666666666,
    "query_memory_kb_total": 84997.33333333333,
    "query_recall@K_total": 0.613157,
    "query_recall@1_total": 0.9944000000000001
  }
}

faiss-ivf-relaxed.json

{
  "metadata": {
    "test_name": "Faiss IVF Relaxed Filter Test",
    "test_id": "Faiss IVF Relaxed Filter Test",
    "date": "05/09/2024 22:19:11",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5219438592 (used) / 26879111168 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 260770.53032599358,
    "delete_model_took_total": 204.0574563240322,
    "delete_index_took_total": 348.1035096532044,
    "train_model_took_total": 1815.439123660326,
    "create_index_took_total": 376.0557709999072,
    "ingest_multi_field_took_total": 190219.80702298848,
    "refresh_index_store_kb_total": 2464222.2392578125,
    "refresh_index_took_total": 484.03268336551264,
    "force_merge_took_total": 5975.129555988436,
    "warmup_operation_took_total": 460.9052030136809,
    "query_with_filter_took_total": 60887.0,
    "query_with_filter_took_p50": 6.0,
    "query_with_filter_took_p90": 7.0,
    "query_with_filter_took_p99": 18.666666666666668,
    "query_with_filter_took_p99.9": 43.0,
    "query_with_filter_took_p100": 537.6666666666666,
    "query_with_filter_client_time_total": 750701.6666666666,
    "query_with_filter_client_time_p50": 74.33333333333333,
    "query_with_filter_client_time_p90": 75.33333333333333,
    "query_with_filter_client_time_p99": 91.0,
    "query_with_filter_client_time_p99.9": 173.33333333333334,
    "query_with_filter_client_time_p100": 716.6666666666666,
    "query_with_filter_memory_kb_total": 1020885.6666666666,
    "query_with_filter_recall@K_total": 0.481398,
    "query_with_filter_recall@1_total": 1.0
  }
}

faiss-ivf-restrictive.json

{
  "metadata": {
    "test_name": "Faiss IVF restrictive Filter Test",
    "test_id": "Faiss IVF restrictive Filter Test",
    "date": "05/09/2024 23:07:35",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5250400256 (used) / 26848280576 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 254351.58079402303,
    "delete_model_took_total": 206.49716966242218,
    "delete_index_took_total": 371.1562816752121,
    "train_model_took_total": 1790.8854183042422,
    "create_index_took_total": 354.80912200485665,
    "ingest_multi_field_took_total": 191448.1718467044,
    "refresh_index_store_kb_total": 2821272.5240885415,
    "refresh_index_took_total": 511.5248623381679,
    "force_merge_took_total": 6363.093521659418,
    "warmup_operation_took_total": 438.10923834098503,
    "query_with_filter_took_total": 52867.333333333336,
    "query_with_filter_took_p50": 5.0,
    "query_with_filter_took_p90": 5.666666666666667,
    "query_with_filter_took_p99": 9.666666666666666,
    "query_with_filter_took_p99.9": 11.333333333333334,
    "query_with_filter_took_p100": 598.6666666666666,
    "query_with_filter_client_time_total": 737961.6666666666,
    "query_with_filter_client_time_p50": 73.66666666666667,
    "query_with_filter_client_time_p90": 74.66666666666667,
    "query_with_filter_client_time_p99": 78.33333333333333,
    "query_with_filter_client_time_p99.9": 176.33333333333334,
    "query_with_filter_client_time_p100": 691.3333333333334,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

lucene-hnsw.json

{
  "metadata": {
    "test_name": "Lucene HNSW",
    "test_id": "Lucene HNSW",
    "date": "05/10/2024 05:22:55",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5211877376 (used) / 26886639616 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 655473.1636070068,
    "delete_index_took_total": 372.35230366544175,
    "create_index_took_total": 349.3909346871078,
    "ingest_took_total": 235346.12898064856,
    "refresh_index_store_kb_total": 2789039.978515625,
    "refresh_index_took_total": 345.89222533395514,
    "force_merge_took_total": 367543.06582933833,
    "query_took_total": 51516.333333333336,
    "query_took_p50": 5.0,
    "query_took_p90": 5.666666666666667,
    "query_took_p99": 8.0,
    "query_took_p99.9": 10.0,
    "query_took_p100": 84.0,
    "query_client_time_total": 712577.3333333334,
    "query_client_time_p50": 71.0,
    "query_client_time_p90": 72.66666666666667,
    "query_client_time_p99": 74.66666666666667,
    "query_client_time_p99.9": 125.0,
    "query_client_time_p100": 368.0,
    "query_memory_kb_total": 0.0,
    "query_recall@K_total": 0.9991249999999999,
    "query_recall@1_total": 1.0
  }
}

lucene-hnsw-nested.json

{
  "metadata": {
    "test_name": "Lucene HNSW Nested Field Test",
    "test_id": "Lucene HNSW Nested Field Test",
    "date": "05/10/2024 04:15:52",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5260668928 (used) / 26829328384 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 804016.3887433102,
    "delete_index_took_total": 400.07745032198727,
    "create_index_took_total": 353.8928006504041,
    "ingest_nested_field_took_total": 364494.8321106688,
    "refresh_index_store_kb_total": 2813402.4329427085,
    "refresh_index_took_total": 384.4405413256027,
    "force_merge_took_total": 380382.9802066709,
    "warmup_operation_took_total": 135.1656336725379,
    "query_nested_field_took_total": 57865.0,
    "query_nested_field_took_p50": 6.0,
    "query_nested_field_took_p90": 6.0,
    "query_nested_field_took_p99": 9.0,
    "query_nested_field_took_p99.9": 11.0,
    "query_nested_field_took_p100": 148.33333333333334,
    "query_nested_field_client_time_total": 719071.0,
    "query_nested_field_client_time_p50": 71.66666666666667,
    "query_nested_field_client_time_p90": 73.33333333333333,
    "query_nested_field_client_time_p99": 75.33333333333333,
    "query_nested_field_client_time_p99.9": 137.0,
    "query_nested_field_client_time_p100": 373.3333333333333,
    "query_nested_field_memory_kb_total": 0.0,
    "query_nested_field_recall@K_total": 0.9991733333333334,
    "query_nested_field_recall@1_total": 1.0
  }
}

lucene-hnsw-relaxed-filter.json

{
  "metadata": {
    "test_name": "Lucene HNSW Relaxed Filter Test",
    "test_id": "Lucene HNSW Relaxed Filter Test",
    "date": "05/10/2024 01:53:02",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5281169408 (used) / 26817314816 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 732562.8504192802,
    "delete_index_took_total": 363.340191989361,
    "create_index_took_total": 349.4545506546274,
    "ingest_multi_field_took_total": 240599.69915998713,
    "refresh_index_store_kb_total": 2817007.5546875,
    "refresh_index_took_total": 393.4494049753994,
    "force_merge_took_total": 407398.2404450071,
    "query_with_filter_took_total": 83458.66666666667,
    "query_with_filter_took_p50": 8.0,
    "query_with_filter_took_p90": 9.666666666666666,
    "query_with_filter_took_p99": 12.0,
    "query_with_filter_took_p99.9": 15.333333333333334,
    "query_with_filter_took_p100": 186.66666666666666,
    "query_with_filter_client_time_total": 739146.0,
    "query_with_filter_client_time_p50": 73.33333333333333,
    "query_with_filter_client_time_p90": 75.66666666666667,
    "query_with_filter_client_time_p99": 78.66666666666667,
    "query_with_filter_client_time_p99.9": 172.0,
    "query_with_filter_client_time_p100": 1129.0,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 0.9993876666666667,
    "query_with_filter_recall@1_total": 1.0
  }
}

lucene-hnsw-restrictive-filter.json

{
  "metadata": {
    "test_name": "Lucene HNSW Restrictive Filter Test",
    "test_id": "Lucene HNSW Restrictive Filter Test",
    "date": "05/10/2024 03:01:25",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "4958932992 (used) / 27139485696 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 690437.9878996458,
    "delete_index_took_total": 388.95499997306615,
    "create_index_took_total": 339.11962966279435,
    "ingest_multi_field_took_total": 265326.39261333196,
    "refresh_index_store_kb_total": 2848035.9166666665,
    "refresh_index_took_total": 372.77007999364287,
    "force_merge_took_total": 362899.7505766844,
    "query_with_filter_took_total": 61111.0,
    "query_with_filter_took_p50": 6.0,
    "query_with_filter_took_p90": 6.333333333333333,
    "query_with_filter_took_p99": 9.333333333333334,
    "query_with_filter_took_p99.9": 11.666666666666666,
    "query_with_filter_took_p100": 150.33333333333334,
    "query_with_filter_client_time_total": 713551.3333333334,
    "query_with_filter_client_time_p50": 71.0,
    "query_with_filter_client_time_p90": 72.66666666666667,
    "query_with_filter_client_time_p99": 75.33333333333333,
    "query_with_filter_client_time_p99.9": 163.0,
    "query_with_filter_client_time_p100": 712.3333333333334,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

nmslib-hnsw.json

{
  "metadata": {
    "test_name": "Nmslib HNSW Test",
    "test_id": "Nmslib HNSW Test",
    "date": "05/10/2024 06:13:47",
    "python_version": "3.8.18 (default, Sep 11 2023, 13:40:15) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.215-181.850.amzn2int.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 16 cores",
    "memory": "5190512640 (used) / 26907938816 (available) / 32570015744 (total)"
  },
  "results": {
    "test_took": 315252.87681130366,
    "delete_index_took_total": 371.37225867869955,
    "create_index_took_total": 352.85584065907943,
    "ingest_took_total": 189281.25270099068,
    "refresh_index_store_kb_total": 3008042.505533854,
    "refresh_index_took_total": 5885.261139657814,
    "force_merge_took_total": 76869.23525631816,
    "warmup_operation_took_total": 430.2329483325593,
    "query_took_total": 42062.666666666664,
    "query_took_p50": 4.0,
    "query_took_p90": 4.666666666666667,
    "query_took_p99": 7.0,
    "query_took_p99.9": 11.666666666666666,
    "query_took_p100": 462.0,
    "query_client_time_total": 719063.3333333334,
    "query_client_time_p50": 71.33333333333333,
    "query_client_time_p90": 73.33333333333333,
    "query_client_time_p99": 75.0,
    "query_client_time_p99.9": 126.66666666666667,
    "query_client_time_p100": 553.3333333333334,
    "query_memory_kb_total": 1278638.6666666667,
    "query_recall@K_total": 0.9988790000000001,
    "query_recall@1_total": 1.0
  }
}

@jmazanec15
Copy link
Member

Client time is higher. Most likely related to client configuration being in diff net. Will look at took times.

@jmazanec15
Copy link
Member

Query P50 and P90

Query times look a little bit different:

image

@jmazanec15
Copy link
Member

Re-running test. There may have been a setup issue. Mimicking setup that was used for 2.13.

@jmazanec15
Copy link
Member

Update: from prelim results, this looks to be an issue with client/cluster configuration. Will post results again once updated.

@jmazanec15
Copy link
Member

image

@jmazanec15
Copy link
Member

results look good

@jmazanec15
Copy link
Member

faiss-hnsw-nested.json

{
"metadata": {
  "test_name": "Faiss HNSW Nested Field Test",
  "test_id": "Faiss HNSW Nested Field Test",
  "date": "05/13/2024 23:34:29",
  "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
  "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
  "processor": "x86_64, 36 cores",
  "memory": "529502208 (used) / 72530317312 (available) / 73743400960 (total)"
},
"results": {
  "test_took": 228920.87416566644,
  "delete_index_took_total": 259.04656766654927,
  "create_index_took_total": 215.27159633342308,
  "ingest_nested_field_took_total": 100829.96140266657,
  "refresh_index_store_kb_total": 2783430.1555989585,
  "refresh_index_took_total": 19161.25362733328,
  "force_merge_took_total": 71349.12648666675,
  "warmup_operation_took_total": 335.8811516665507,
  "query_nested_field_took_total": 36770.333333333336,
  "query_nested_field_took_p50": 3.6666666666666665,
  "query_nested_field_took_p90": 4.0,
  "query_nested_field_took_p99": 4.666666666666667,
  "query_nested_field_took_p99.9": 11.0,
  "query_nested_field_took_p100": 792.6666666666666,
  "query_nested_field_client_time_total": 73109.33333333333,
  "query_nested_field_client_time_p50": 7.0,
  "query_nested_field_client_time_p90": 8.0,
  "query_nested_field_client_time_p99": 8.666666666666666,
  "query_nested_field_client_time_p99.9": 44.0,
  "query_nested_field_client_time_p100": 799.3333333333334,
  "query_nested_field_memory_kb_total": 1297460.0,
  "query_nested_field_recall@K_total": 0.9991859999999999,
  "query_nested_field_recall@1_total": 1.0
}
}

faiss-hnsw-relaxed-filter.json

{
"metadata": {
  "test_name": "Faiss HNSW Relaxed Filter Test",
  "test_id": "Faiss HNSW Relaxed Filter Test",
  "date": "05/13/2024 23:07:43",
  "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
  "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
  "processor": "x86_64, 36 cores",
  "memory": "552996864 (used) / 72507604992 (available) / 73743400960 (total)"
},
"results": {
  "test_took": 211080.95865599994,
  "delete_index_took_total": 192.80809499999427,
  "create_index_took_total": 302.1410496666401,
  "ingest_multi_field_took_total": 90617.40822166663,
  "refresh_index_store_kb_total": 3039407.9401041665,
  "refresh_index_took_total": 16333.819989666685,
  "force_merge_took_total": 62734.52694399995,
  "warmup_operation_took_total": 359.9210226667158,
  "query_with_filter_took_total": 40540.333333333336,
  "query_with_filter_took_p50": 4.0,
  "query_with_filter_took_p90": 5.0,
  "query_with_filter_took_p99": 5.666666666666667,
  "query_with_filter_took_p99.9": 19.0,
  "query_with_filter_took_p100": 643.3333333333334,
  "query_with_filter_client_time_total": 75867.33333333333,
  "query_with_filter_client_time_p50": 7.333333333333333,
  "query_with_filter_client_time_p90": 8.333333333333334,
  "query_with_filter_client_time_p99": 9.666666666666666,
  "query_with_filter_client_time_p99.9": 64.66666666666667,
  "query_with_filter_client_time_p100": 702.3333333333334,
  "query_with_filter_memory_kb_total": 1297436.0,
  "query_with_filter_recall@K_total": 0.9990813333333333,
  "query_with_filter_recall@1_total": 1.0
}
}

faiss-hnsw-restrictive-filter.json

  {
  "metadata": {
    "test_name": "Faiss HNSW Restrictive Filter Test",
    "test_id": "Faiss HNSW Restrictive Filter Test",
    "date": "05/13/2024 23:19:37",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "559210496 (used) / 72501354496 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 177184.75661999997,
    "delete_index_took_total": 236.05446966674512,
    "create_index_took_total": 232.9487979999764,
    "ingest_multi_field_took_total": 59100.66246100003,
    "refresh_index_store_kb_total": 3040646.8118489585,
    "refresh_index_took_total": 12782.6524783333,
    "force_merge_took_total": 72384.99885399999,
    "warmup_operation_took_total": 354.7728923332822,
    "query_with_filter_took_total": 32092.666666666668,
    "query_with_filter_took_p50": 3.0,
    "query_with_filter_took_p90": 3.6666666666666665,
    "query_with_filter_took_p99": 4.333333333333333,
    "query_with_filter_took_p99.9": 14.333333333333334,
    "query_with_filter_took_p100": 444.0,
    "query_with_filter_client_time_total": 68110.0,
    "query_with_filter_client_time_p50": 6.666666666666667,
    "query_with_filter_client_time_p90": 7.333333333333333,
    "query_with_filter_client_time_p99": 8.333333333333334,
    "query_with_filter_client_time_p99.9": 58.666666666666664,
    "query_with_filter_client_time_p100": 544.0,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

faiss-hnsw.json

{
"metadata": {
  "test_name": "Faiss HNSW Test",
  "test_id": "Faiss HNSW Test",
  "date": "05/13/2024 23:47:03",
  "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
  "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
  "processor": "x86_64, 36 cores",
  "memory": "495321088 (used) / 72563970048 (available) / 73743400960 (total)"
},
"results": {
  "test_took": 195362.7564463336,
  "delete_index_took_total": 357.57908333334854,
  "create_index_took_total": 219.8324743330886,
  "ingest_took_total": 88968.6990236669,
  "refresh_index_store_kb_total": 2964066.5361328125,
  "refresh_index_took_total": 16390.85980000012,
  "force_merge_took_total": 62780.24719133342,
  "warmup_operation_took_total": 337.5388736667446,
  "query_took_total": 26308.0,
  "query_took_p50": 3.0,
  "query_took_p90": 3.0,
  "query_took_p99": 4.333333333333333,
  "query_took_p99.9": 6.333333333333333,
  "query_took_p100": 529.3333333333334,
  "query_client_time_total": 57778.333333333336,
  "query_client_time_p50": 5.333333333333333,
  "query_client_time_p90": 6.0,
  "query_client_time_p99": 7.666666666666667,
  "query_client_time_p99.9": 43.666666666666664,
  "query_client_time_p100": 535.0,
  "query_memory_kb_total": 1297436.0,
  "query_recall@K_total": 0.9992353333333334,
  "query_recall@1_total": 1.0
}
}

faiss-hnswpq.json

{
  "metadata": {
    "test_name": "Faiss HNSW PQ Test",
    "test_id": "Faiss HNSW PQ Test",
    "date": "05/13/2024 23:57:37",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "468287488 (used) / 72590540800 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 150485.97180566684,
    "delete_model_took_total": 73.05389799997404,
    "delete_index_took_total": 276.4082876668302,
    "train_model_took_total": 16115.691309333366,
    "create_index_took_total": 199.9784133331559,
    "ingest_took_total": 56701.77150233334,
    "refresh_index_store_kb_total": 1853516.6363932292,
    "refresh_index_took_total": 8028.591848666717,
    "force_merge_took_total": 46641.21499466683,
    "warmup_operation_took_total": 103.92821833329435,
    "query_took_total": 22345.333333333332,
    "query_took_p50": 2.0,
    "query_took_p90": 2.6666666666666665,
    "query_took_p99": 3.6666666666666665,
    "query_took_p99.9": 5.333333333333333,
    "query_took_p100": 798.3333333333334,
    "query_client_time_total": 57834.333333333336,
    "query_client_time_p50": 5.666666666666667,
    "query_client_time_p90": 6.0,
    "query_client_time_p99": 7.333333333333333,
    "query_client_time_p99.9": 41.666666666666664,
    "query_client_time_p100": 804.3333333333334,
    "query_memory_kb_total": 334828.0,
    "query_recall@K_total": 0.6291350000000001,
    "query_recall@1_total": 0.9947
  }
}

faiss-ivf-relaxed.json

{
  "metadata": {
    "test_name": "Faiss IVF Relaxed Filter Test",
    "test_id": "Faiss IVF Relaxed Filter Test",
    "date": "05/14/2024 00:05:58",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "559128576 (used) / 72499691520 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 106409.99804533391,
    "delete_model_took_total": 118.77878866683507,
    "delete_index_took_total": 351.4939836668418,
    "train_model_took_total": 1419.0895883333117,
    "create_index_took_total": 208.78923233340174,
    "ingest_multi_field_took_total": 58199.80494033341,
    "refresh_index_store_kb_total": 2053612.6458333333,
    "refresh_index_took_total": 352.19210166633275,
    "force_merge_took_total": 5900.331610000042,
    "warmup_operation_took_total": 331.5178003337375,
    "query_with_filter_took_total": 39528.0,
    "query_with_filter_took_p50": 4.0,
    "query_with_filter_took_p90": 4.666666666666667,
    "query_with_filter_took_p99": 14.333333333333334,
    "query_with_filter_took_p99.9": 27.333333333333332,
    "query_with_filter_took_p100": 588.0,
    "query_with_filter_client_time_total": 76202.33333333333,
    "query_with_filter_client_time_p50": 7.333333333333333,
    "query_with_filter_client_time_p90": 8.333333333333334,
    "query_with_filter_client_time_p99": 18.333333333333332,
    "query_with_filter_client_time_p99.9": 53.666666666666664,
    "query_with_filter_client_time_p100": 653.0,
    "query_with_filter_memory_kb_total": 1050030.0,
    "query_with_filter_recall@K_total": 0.4770486666666667,
    "query_with_filter_recall@1_total": 0.9998999999999999
  }
}

faiss-ivf-restrictive.json

{
  "metadata": {
    "test_name": "Faiss IVF restrictive Filter Test",
    "test_id": "Faiss IVF restrictive Filter Test",
    "date": "05/14/2024 00:14:05",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "561737728 (used) / 72497094656 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 100598.04578699957,
    "delete_model_took_total": 69.23975966644018,
    "delete_index_took_total": 227.05521799980488,
    "train_model_took_total": 1346.4517510004346,
    "create_index_took_total": 220.84387533353342,
    "ingest_multi_field_took_total": 58690.59310433295,
    "refresh_index_store_kb_total": 2447255.9303385415,
    "refresh_index_took_total": 415.18830566656106,
    "force_merge_took_total": 5974.611246999909,
    "warmup_operation_took_total": 321.3958593332791,
    "query_with_filter_took_total": 33332.666666666664,
    "query_with_filter_took_p50": 3.0,
    "query_with_filter_took_p90": 4.0,
    "query_with_filter_took_p99": 5.0,
    "query_with_filter_took_p99.9": 10.666666666666666,
    "query_with_filter_took_p100": 832.6666666666666,
    "query_with_filter_client_time_total": 70503.66666666667,
    "query_with_filter_client_time_p50": 6.666666666666667,
    "query_with_filter_client_time_p90": 7.666666666666667,
    "query_with_filter_client_time_p99": 9.0,
    "query_with_filter_client_time_p99.9": 52.666666666666664,
    "query_with_filter_client_time_p100": 838.0,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

faiss-ivf.json

{
  "metadata": {
    "test_name": "Faiss IVF",
    "test_id": "Faiss IVF",
    "date": "05/14/2024 00:21:09",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "464719872 (used) / 72594100224 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 85450.95730533342,
    "delete_model_took_total": 67.1353940000093,
    "delete_index_took_total": 266.10771133346134,
    "train_model_took_total": 1411.311601333182,
    "create_index_took_total": 202.90694899995287,
    "ingest_took_total": 55441.79024966676,
    "refresh_index_store_kb_total": 1703463.1578776042,
    "refresh_index_took_total": 300.80459833334316,
    "force_merge_took_total": 5112.753151000106,
    "warmup_operation_took_total": 310.14765066659794,
    "query_took_total": 22338.0,
    "query_took_p50": 2.0,
    "query_took_p90": 3.0,
    "query_took_p99": 3.0,
    "query_took_p99.9": 7.666666666666667,
    "query_took_p100": 486.3333333333333,
    "query_client_time_total": 52932.0,
    "query_client_time_p50": 5.0,
    "query_client_time_p90": 6.0,
    "query_client_time_p99": 6.0,
    "query_client_time_p99.9": 47.0,
    "query_client_time_p100": 507.6666666666667,
    "query_memory_kb_total": 1050030.0,
    "query_recall@K_total": 0.9471383333333333,
    "query_recall@1_total": 1.0
  }
}

faiss-ivfpq.json

{
  "metadata": {
    "test_name": "Faiss IVF PQ Test",
    "test_id": "Faiss IVF PQ Test",
    "date": "05/14/2024 00:29:06",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "467275776 (used) / 72591548416 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 100744.50930133408,
    "delete_model_took_total": 70.10026866707146,
    "delete_index_took_total": 285.5522106665376,
    "train_model_took_total": 16596.928837666685,
    "create_index_took_total": 209.56706066681363,
    "ingest_took_total": 54639.90690399988,
    "refresh_index_store_kb_total": 1651325.3912760417,
    "refresh_index_took_total": 504.27526833330677,
    "force_merge_took_total": 6060.784723000324,
    "warmup_operation_took_total": 266.72736166680505,
    "query_took_total": 22110.666666666668,
    "query_took_p50": 2.0,
    "query_took_p90": 2.6666666666666665,
    "query_took_p99": 3.0,
    "query_took_p99.9": 4.666666666666667,
    "query_took_p100": 694.0,
    "query_client_time_total": 54919.0,
    "query_client_time_p50": 5.333333333333333,
    "query_client_time_p90": 6.0,
    "query_client_time_p99": 6.666666666666667,
    "query_client_time_p99.9": 41.333333333333336,
    "query_client_time_p100": 699.3333333333334,
    "query_memory_kb_total": 87428.0,
    "query_recall@K_total": 0.613616,
    "query_recall@1_total": 0.9945
  }
}

lucene-hnsw-nested.json

{
  "metadata": {
    "test_name": "Lucene HNSW Nested Field Test",
    "test_id": "Lucene HNSW Nested Field Test",
    "date": "05/14/2024 01:33:50",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "540360704 (used) / 72518348800 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 392133.4661409997,
    "delete_index_took_total": 249.50688266653742,
    "create_index_took_total": 183.82586733241624,
    "ingest_nested_field_took_total": 215552.41006633334,
    "refresh_index_store_kb_total": 2820329.0162760415,
    "refresh_index_took_total": 467.502833334341,
    "force_merge_took_total": 130820.96212733387,
    "warmup_operation_took_total": 20.92503066584565,
    "query_nested_field_took_total": 44838.333333333336,
    "query_nested_field_took_p50": 4.0,
    "query_nested_field_took_p90": 5.333333333333333,
    "query_nested_field_took_p99": 7.666666666666667,
    "query_nested_field_took_p99.9": 9.0,
    "query_nested_field_took_p100": 900.0,
    "query_nested_field_client_time_total": 80429.66666666667,
    "query_nested_field_client_time_p50": 7.666666666666667,
    "query_nested_field_client_time_p90": 9.0,
    "query_nested_field_client_time_p99": 11.333333333333334,
    "query_nested_field_client_time_p99.9": 48.0,
    "query_nested_field_client_time_p100": 905.6666666666666,
    "query_nested_field_memory_kb_total": 0.0,
    "query_nested_field_recall@K_total": 0.9991396666666668,
    "query_nested_field_recall@1_total": 1.0
  }
}

lucene-hnsw-relaxed-filter.json

{
  "metadata": {
    "test_name": "Lucene HNSW Relaxed Filter Test",
    "test_id": "Lucene HNSW Relaxed Filter Test",
    "date": "05/14/2024 00:50:46",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "573599744 (used) / 72485158912 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 377216.2145319991,
    "delete_index_took_total": 258.5346459997406,
    "create_index_took_total": 182.63600066651028,
    "ingest_multi_field_took_total": 131449.50534633306,
    "refresh_index_store_kb_total": 1930048.6614583333,
    "refresh_index_took_total": 456.41236900003906,
    "force_merge_took_total": 176472.45950333308,
    "query_with_filter_took_total": 68396.66666666667,
    "query_with_filter_took_p50": 6.0,
    "query_with_filter_took_p90": 7.666666666666667,
    "query_with_filter_took_p99": 12.666666666666666,
    "query_with_filter_took_p99.9": 16.333333333333332,
    "query_with_filter_took_p100": 1054.0,
    "query_with_filter_client_time_total": 100378.0,
    "query_with_filter_client_time_p50": 9.0,
    "query_with_filter_client_time_p90": 11.0,
    "query_with_filter_client_time_p99": 15.666666666666666,
    "query_with_filter_client_time_p99.9": 190.33333333333334,
    "query_with_filter_client_time_p100": 1058.3333333333333,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 0.9993426666666667,
    "query_with_filter_recall@1_total": 1.0
  }
}

lucene-hnsw-restrictive-filter.json

{
  "metadata": {
    "test_name": "Lucene HNSW Restrictive Filter Test",
    "test_id": "Lucene HNSW Restrictive Filter Test",
    "date": "05/14/2024 01:11:18",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "562434048 (used) / 72496336896 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 349807.0192546669,
    "delete_index_took_total": 284.1160946666908,
    "create_index_took_total": 203.6125216667036,
    "ingest_multi_field_took_total": 136634.37234533374,
    "refresh_index_store_kb_total": 1952388.5205078125,
    "refresh_index_took_total": 430.03998233325547,
    "force_merge_took_total": 162792.21164399982,
    "query_with_filter_took_total": 49462.666666666664,
    "query_with_filter_took_p50": 4.0,
    "query_with_filter_took_p90": 5.0,
    "query_with_filter_took_p99": 7.0,
    "query_with_filter_took_p99.9": 13.0,
    "query_with_filter_took_p100": 1046.3333333333333,
    "query_with_filter_client_time_total": 86624.0,
    "query_with_filter_client_time_p50": 7.666666666666667,
    "query_with_filter_client_time_p90": 8.666666666666666,
    "query_with_filter_client_time_p99": 11.0,
    "query_with_filter_client_time_p99.9": 197.66666666666666,
    "query_with_filter_client_time_p100": 1050.3333333333333,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

lucene-hnsw.json

{
  "metadata": {
    "test_name": "Lucene HNSW",
    "test_id": "Lucene HNSW",
    "date": "05/14/2024 01:53:54",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "477655040 (used) / 72581033984 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 345192.0747286663,
    "delete_index_took_total": 290.1537996664653,
    "create_index_took_total": 200.91187533337992,
    "ingest_took_total": 130954.66350966731,
    "refresh_index_store_kb_total": 2355241.466471354,
    "refresh_index_took_total": 396.26103066621,
    "force_merge_took_total": 169152.75117999956,
    "query_took_total": 44197.333333333336,
    "query_took_p50": 4.0,
    "query_took_p90": 4.0,
    "query_took_p99": 5.0,
    "query_took_p99.9": 15.0,
    "query_took_p100": 1039.0,
    "query_client_time_total": 76474.33333333333,
    "query_client_time_p50": 6.666666666666667,
    "query_client_time_p90": 7.333333333333333,
    "query_client_time_p99": 9.0,
    "query_client_time_p99.9": 245.0,
    "query_client_time_p100": 1043.0,
    "query_memory_kb_total": 0.0,
    "query_recall@K_total": 0.9991003333333334,
    "query_recall@1_total": 1.0
  }
}

nmslib-hnsw.json

{
  "metadata": {
    "test_name": "Nmslib HNSW Test",
    "test_id": "Nmslib HNSW Test",
    "date": "05/14/2024 02:04:48",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "502788096 (used) / 72555819008 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 162413.8357240008,
    "delete_index_took_total": 341.76358333328,
    "create_index_took_total": 192.5681643333519,
    "ingest_took_total": 56901.3385620001,
    "refresh_index_store_kb_total": 2760467.2431640625,
    "refresh_index_took_total": 14015.495272666763,
    "force_merge_took_total": 64194.064072001005,
    "warmup_operation_took_total": 289.2727363329565,
    "query_took_total": 26479.333333333332,
    "query_took_p50": 2.6666666666666665,
    "query_took_p90": 3.0,
    "query_took_p99": 3.6666666666666665,
    "query_took_p99.9": 8.0,
    "query_took_p100": 495.0,
    "query_client_time_total": 58053.333333333336,
    "query_client_time_p50": 5.333333333333333,
    "query_client_time_p90": 6.0,
    "query_client_time_p99": 7.0,
    "query_client_time_p99.9": 52.333333333333336,
    "query_client_time_p100": 500.3333333333333,
    "query_memory_kb_total": 1305860.6666666667,
    "query_recall@K_total": 0.9988543333333334,
    "query_recall@1_total": 1.0
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jmazanec15 jmazanec15

Labels
v2.14.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jmazanec15