Release Version 2.5.0

[peterzhuamazon]

Release Version 2.5.0

This is a component issue for 2.5.0.
Coming from opensearch-build#2908. Please follow the following checklist.
Please refer to the DATES / CAMPAIGNS in that post.

How to use this issue

This Component Release Issue

This issue captures the state of the OpenSearch release, on component/plugin level; its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help.
Any release related work can be linked to this issue or added as comments to create visiblity into the release status.

Release Steps

There are several steps to the release process; these steps are completed as the whole component release and components that are behind present risk to the release. The component owner resolves the tasks in this issue and communicate with the overall release owner to make sure each component are moving along as expected.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release. The most current set of dates is on the overall release issue linked at the top of this issue.

The Overall Release Issue

Linked at the top of this issue, the overall release issue captures the state of the entire OpenSearch release including references to this issue, the release owner which is the assignee is responsible for communicating the release status broadly. Please contact them or @mention them on that issue for help.

What should I do if my plugin isn't making any changes?

If including changes in this release, increment the version on 2.5 branch to 2.5.0 for Min/Core, and 2.5.0.0 for components. Otherwise, keep the version number unchanged for both.

Preparation

• Assign this issue to a release owner.
• Finalize scope and feature set and update the Public Roadmap.
• All the tasks in this issue have been reviewed by the release owner.
• Create, update, triage and label all features and issues targeted for this release with v2.5.0.
• Cut 2.5 branch

CI/CD

• All code changes for 2.5.0 are complete.
• Ensure working and passing CI.
• Check that this repo is included in the distribution manifest.

Pre-Release

• Update to the 2.5.0 release branch in the distribution manifest.
• Increment the version on the parent branch to the next development iteration.
• Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
• Confirm that all changes for 2.5.0 have been merged.
• Add this repo to the manifest for the next developer iteration.

Release Testing

• Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
• Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
• Sanity Testing: Sanity testing and fixing of critical issues found.
• File issues for all intermittent test failures.

Release

• Complete documentation.
• Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

• Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
• Add this repo to the manifest of the next eventual security patch version.
• Suggest improvements to this template.
• Conduct a retrospective, and publish its results.

[jmazanec15]

Performance Results

Run on sift 1M data set

Query latency

Ingest took time

Conclusions

It appears faiss/nmslib query latencies have increased. Will investigate to see if this is noise or indicative of something more.

Raw Results

lucene-hnsw

{
  "metadata": {
    "test_name": "index-workflow",
    "test_id": "Index workflow",
    "date": "01/14/2023 00:38:56",
    "python_version": "3.8.15 (default, Nov  4 2022, 20:59:55) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.147-133.644.amzn2.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 36 cores",
    "memory": "388161536 (used) / 72579211264 (available) / 73639055360 (total)"
  },
  "results": {
    "test_took": 255607.83097269997,
    "delete_index_took_total": 571.5781850999747,
    "create_index_took_total": 275.91594969992457,
    "ingest_took_total": 164933.61317500006,
    "refresh_index_store_kb_total": 2399878.1208984377,
    "refresh_index_took_total": 526.2236628999972,
    "query_took_total": 89300.5,
    "query_took_p50": 9.0,
    "query_took_p90": 10.3,
    "query_took_p99": 11.3,
    "query_memory_kb_total": 0.0,
    "query_recall@K_total": 0.9996663,
    "query_recall@1_total": 1.0
  }
}

nmslib-hnsw

{
  "metadata": {
    "test_name": "index-workflow",
    "test_id": "Index workflow",
    "date": "01/17/2023 18:21:49",
    "python_version": "3.8.15 (default, Nov  4 2022, 20:59:55) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.147-133.644.amzn2.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 36 cores",
    "memory": "518090752 (used) / 72410464256 (available) / 73639055360 (total)"
  },
  "results": {
    "test_took": 132904.74214252457,
    "delete_index_took_total": 643.764022202231,
    "create_index_took_total": 205.00653621274978,
    "ingest_took_total": 57379.42679281114,
    "refresh_index_store_kb_total": 2974269.7612304688,
    "refresh_index_took_total": 14803.044791298453,
    "query_took_total": 59873.5,
    "query_took_p50": 6.0,
    "query_took_p90": 6.7,
    "query_took_p99": 8.0,
    "query_memory_kb_total": 1305907.3,
    "query_recall@K_total": 0.9997376000000001,
    "query_recall@1_total": 1.0
  }
}

faiss-ivf

{
  "metadata": {
    "test_name": "index-workflow",
    "test_id": "index workflow",
    "date": "01/17/2023 19:32:11",
    "python_version": "3.8.15 (default, Nov  4 2022, 20:59:55) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.147-133.644.amzn2.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 36 cores",
    "memory": "519663616 (used) / 72408895488 (available) / 73639055360 (total)"
  },
  "results": {
    "test_took": 147740.67420917127,
    "delete_model_took_total": 76.34547710767947,
    "delete_index_took_total": 777.1025589958299,
    "train_model_took_total": 1565.2957786049228,
    "create_index_took_total": 212.9984110710211,
    "ingest_took_total": 55251.11392940162,
    "refresh_index_store_kb_total": 2474974.2994140624,
    "refresh_index_took_total": 350.6180539901834,
    "query_took_total": 89507.2,
    "query_took_p50": 8.9,
    "query_took_p90": 9.2,
    "query_took_p99": 12.6,
    "query_memory_kb_total": 1074782.6,
    "query_recall@K_total": 0.9471395000000001,
    "query_recall@1_total": 1.0
  }
}

faiss-ivfpq

{
  "metadata": {
    "test_name": "index-workflow",
    "test_id": "index workflow",
    "date": "01/17/2023 21:57:20",
    "python_version": "3.8.15 (default, Nov  4 2022, 20:59:55) \n[GCC 11.2.0]",
    "os_version": "Linux-5.10.147-133.644.amzn2.x86_64-x86_64-with-glibc2.17",
    "processor": "x86_64, 36 cores",
    "memory": "526671872 (used) / 72401801216 (available) / 73639055360 (total)"
  },
  "results": {
    "test_took": 197083.63207730983,
    "delete_model_took_total": 74.28844840615056,
    "delete_index_took_total": 713.4049198997673,
    "train_model_took_total": 54554.71392819891,
    "create_index_took_total": 202.24073010031134,
    "ingest_took_total": 54378.95677300403,
    "refresh_index_store_kb_total": 1723524.0901367187,
    "refresh_index_took_total": 1824.3272777006496,
    "query_took_total": 85335.7,
    "query_took_p50": 8.3,
    "query_took_p90": 9.7,
    "query_took_p99": 12.2,
    "query_memory_kb_total": 146438.9,
    "query_recall@K_total": 0.613657,
    "query_recall@1_total": 0.9942999999999997
  }
}

[jmazanec15]

After some more tests, I do not believe this is a true degradation. The source of the discrepancy is that the 2.5 cluster produces more segments than 2.4 cluster with a refresh interval of 1s.

To prove this, I ran the indexing workload and tracked how many segments are produced for nmslib-hnsw and faiss-ivfpq:

	2.4	2.5
nmslib hnsw segment count - test 1	243	295
nmslib hnsw segment count - test 2	247	267
nmslib hnsw segment count - test 3	237	284
nmslib hnsw segment count - avg	242.33333	282
faiss ivfpq segment count - test 1	368	446
faiss ivfpq segment count - test 2	406	489
faiss ivfpq segment count - test 3	384	459
faiss ivfpq segment count - avg	386	464.66667

From here, I ran a set of experiments where the refresh interval is set to 30s. This produces the following graph:

Its unclear at the moment what is causing more segments to be created with 2.5 than 2.4. Potentially, segment creation is faster in 2.5, allowing the 1s refresh interval to be followed closer than 2.4 (i.e. faster indexing prevents segment creation log jam). Additionally, for future releases, we need to figure out how to prevent discrepancies based on segment count in our performance tests. We can take this as an AI.

[navneet1v]

for future releases, we need to figure out how to prevent discrepancies based on segment count in our performance tests. We can take this as an AI.

Can we add force merge segments steps in the perf tool and put a fixed value for segment count to remove this discrepancy? Won't that solve the issue?

POST /my-index/_forcemerge?max_num_segments=<desired-number-of-segments>