[Searchable Snapshot] Configure and run initial benchmarks #4797

andrross · 2022-10-14T18:46:35Z

The goal of this task is to get some update the benchmark tooling and get some initial benchmarks of the searchable snapshot feature. I believe many of the opensearch-benchmark workloads involve discrete phases like: create index->index documents->run query load. Ideally we can augment/extend some of these workloads for searchable snapshots where it would be: create index->index documents->take snapshot->delete local index->create searchable snapshot index->run query load. We could then get a good performance comparison between local disk versus remote searchable snapshot storage for a given workload.

The goal of this task is to get initial numbers for searchable snapshots (which will be bad until performance optimizations are completed) as well as create a plan for adding first class support for testing searchable snapshot workloads to opensearch-benchmark.

tlfeng · 2022-10-25T07:11:33Z

[Update on 11/13/2022]
Outcome of the issue (mentioned in a below comment #4797 (comment)):
Initial benchmark result: #4797 (comment)
Plan to support running benchmark for the feature directly: #4797 (comment)

Did a very initial benchmark for Searchable Snapshot feature.
I setup an OpenSearch node by myself and use opensearch-benchmark tool to run workload on the custom cluster, the reason not to use a node provisioned by the benchmark tool is, there are some custom configuration to an OpenSearch cluster to use the new feature.

Node configuration:
The OpenSearch distribution is built from current 2.x branch (https://github.com/opensearch-project/OpenSearch/tree/5482d9d209b1f6b00c67169a5eff9e361c4964a7)
1 Add path.repo: "/tmp/opensearch_backups" to config/opensearch.yml to use local file system to store snapshots.
2 Add -Dopensearch.experimental.feature.searchable_snapshot.enabled=true to jvm.options to enable the new feature.

Benchmark workload configuration:
1 Create a folder to hold the workload files
1 Prepare testing data: index mapping file index.json and the index data file documents.json
2 Write custom workload definition: (below is what I used to benchmark searching index that backed by snapshot)
It includes steps: delete-index-before-test, create-index, check-cluster-health, create-snapshot-repository, create-snapshot, wait-for-snapshot-create, delete-local-index-to-create-from-snapshot, restore-snapshot, force-merge, query-match-all

{
  "version": 2,
  "description": "Initial benchmark for Searchable Snapshot feature",
  "indices": [
    {
      "name": "geonames",
      "body": "index.json"
    }
  ],
  "corpora": [
    {
      "name": "sample-document",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 11658903,
          "uncompressed-bytes": 1544799789
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "name": "delete-index-before-test",
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "green"
        },
        "retry-until-success": true
      }
    },
    {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": 5000
      },
      "warmup-time-period": 120,
      "clients": 8
    },
    {
      "operation": {
        "operation-type": "create-snapshot-repository",
        "repository": "test-repository",
        "body": {
          "type": "fs",
          "settings": {
            "location": "/tmp/opensearch_backups"
          }
        }
      }
    },
    {
      "operation": {
        "operation-type": "create-snapshot",
        "repository": "test-repository",
        "snapshot": "test-snapshot",
        "body": {
          "indices": []
        }
      }
    },
    {
      "operation": {
        "operation-type": "wait-for-snapshot-create",
        "repository": "test-repository",
        "snapshot": "test-snapshot"
      }
    },
    {
      "operation": {
        "name": "delete-local-index-to-create-from-snapshot",
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "restore-snapshot",
        "repository": "test-repository",
        "snapshot": "test-snapshot",
        "body": {
          "indices": [],
          "storage_type": "remote_snapshot"
        },
        "wait-for-completion": true,
        "request-params": {
          "request_timeout": 7200
        }
      }
    },
    {
      "operation": {
        "operation-type": "force-merge"
      }
    },
    {
      "operation": {
        "name": "query-match-all",
        "operation-type": "search",
        "body": {
          "query": {
            "match_all": {}
          }
        }
      },
      "clients": 8,
      "warmup-iterations": 1000,
      "iterations": 1000,
      "target-throughput": 100
    }
  ]
}

3 The workload can be modified to a control group by removing the operations of deleting index and restoring from snapshot.

Run benchmark:
1 Start the OpenSearch node
2 Run benchmark in the node: opensearch-benchmark execute_test --workload-path=<path to the workload folder> --target-hosts=127.0.0.1:9200 --pipeline=benchmark-only (--test-mode --kill-running-processes)
3 Need to remove data folder of the node and the snapshot folder before running again.

Although I didn't find difference in the search performance between the local index and index backed by snapshot, the procedure in above can be taken as a reference.

andrross · 2022-10-27T20:13:09Z

Great work @tlfeng! One nitpick...that force-merge step should not be there after creating the searchable snapshot index (it should actually fail with the latest code). Also, can you re-run this with an S3 repository instead of a file system repository? I think everything will be the same except for the initial cluster setup.

tlfeng · 2022-10-31T05:45:48Z

After discovery and lots of experiments, I'm able to make the benchmark runs with a cluster provisioned by the benchmarking tool, instead of an external cluster set up manually.

[Updated on 11/10/2022]
Please completely refer to my below comment #4797 (comment) to configure the benchmark tool and run the performance test for Searchable Snapshots feature.
Here are some additional usage of OpenSearch-Benchmark, found during my discovery.

Create custom OpenSearch node configuration for the tool
Although there are some default config options which located in https://github.com/opensearch-project/opensearch-benchmark/tree/0.1.0/osbenchmark/resources/provision_configs, the following steps aim to create a custom configuration, such as defining opensearch.yml and jvm.options

Take creating a custom node configuration for Searchable Snapshot as an example:

Create a new directory in a folder with the following structure (the name "provision_configs" can be changed):

.
└── provision_configs
    └── provision_config_instances
        └── v1
            ├── searchable-snapshot
            │   └── templates
            │       └── config
            │           └── jvm.options
            ├── searchable-snapshot.ini

Content of searchable-snapshot.ini: (type=mixin is the only value need to be the same)

[meta]
description=Node configuration for testing Searchable Snapshots
type=mixin

[config]
base=searchable-snapshot

Content of jvm.options

-Dopensearch.experimental.feature.searchable_snapshot.enabled=true

(if you want to assign the "access key" and "secret key" of AWS account in workload files directly, append -Dopensearch.allow_insecure_settings=true)

Copy defaults.ini (or 8gheap.ini or 16gheap.ini when want a larger Java heap size) and vanilla folder from https://github.com/opensearch-project/opensearch-benchmark/tree/0.1.0/osbenchmark/resources/provision_configs/main/provision_config_instances/v1 to your own v1 folder.
Copy core-plugins.txt and repository_s3 folder from https://github.com/opensearch-project/opensearch-benchmark/tree/0.1.0/osbenchmark/resources/provision_configs/main/plugins/v1 to your own provision_configs folder. The folder structure will be:

.
└── provision_configs
    └── provision_config_instances
    │   └── v1
    │       └── ...
    └── plugins
        └── v1
            └── ...

To use this custom configuration, add --provision-config-path=~/Documents/searchable_snapshot_workload/provision_configs --provision-config-instance="defaults,searchable-snapshot" to opensearch-benchmark command

The custom workload I used to run a simple benchmark
Content of the file workload.json:
The single workload file can be used for both experimental group and control group by filtering the task to be executed.

{
  "version": 2,
  "description": "Initial benchmark for Searchable Snapshot feature",
  "indices": [
    {
      "name": "geonames",
      "body": "index.json"
    }
  ],
  "corpora": [
    {
      "name": "sample-document",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 12349803,
          "uncompressed-bytes": 1637903815
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "name": "delete-index-before-test",
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "green"
        },
        "retry-until-success": true
      }
    },
    {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": 5000
      },
      "warmup-time-period": 120,
      "clients": 8
    },
    {
      "operation": {
        "operation-type": "create-snapshot-repository",
        "repository": "test-repository",
        "body": {
          "type": "s3",
          "settings": {
            "bucket": "<Your bucket name",
            "region": "<Your bucket region>"
          }
        }
      }
    },
    {
      "operation": {
        "operation-type": "create-snapshot",
        "repository": "test-repository",
        "snapshot": "{{ snapshot_name | default('test-snapshot') }}",
        "body": {
          "indices": []
        }
      }
    },
    {
      "operation": {
        "operation-type": "wait-for-snapshot-create",
        "repository": "test-repository",
        "snapshot": "{{ snapshot_name | default('test-snapshot') }}"
      }
    },
    {
      "operation": {
        "name": "delete-local-index-to-create-from-snapshot",
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "restore-snapshot",
        "repository": "test-repository",
        "snapshot": "{{ snapshot_name | default('test-snapshot') }}",
        "body": {
          "indices": [],
          "storage_type": "remote_snapshot"
        },
        "wait-for-completion": true,
        "request-params": {
          "request_timeout": 7200
        }
      }
    },
    {
      "operation": {
        "operation-type": "force-merge"
      }
    },
    {
      "operation": {
        "name": "query-match-all",
        "operation-type": "search",
        "body": {
          "query": {
            "match_all": {}
          }
        }
      },
      "clients": 8,
      "warmup-iterations": 1000,
      "iterations": 1000,
      "target-throughput": 100
    }
  ]
}

Note for workload.json file:

If added -Dopensearch.allow_insecure_settings=true in jvm.options, AWS account credentials can be written in this file:

{
      "operation": {
        "operation-type": "create-snapshot-repository",
        "repository": "test-repository",
        "body": {
          "type": "s3",
          "settings": {
            "bucket": "<Your bucket name>",
            "region": "<Your bucket region>",
            "access_key": "<your AWS access key>",
            "secret_key": "<your AWS secret key>"
          }
        }
      }
    },

The reason to set snapshot_name as a parameter is, there is no operation defined in benchmarking tool to delete a snapshot, to be convenient for running the benchmark multiple times without manually delete the old snapshots, make it as a parameter can assign a different name to the snapshot easier.
To use an existing data set from the repository https://github.com/opensearch-project/opensearch-benchmark-workloads to run the benchmark, take nyc_taxis as an example:
Download the index.json file from the git repository (location https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/ae11cf74873bff91c7299257f1fbcb3cdde47987/nyc_taxis/index.json), put it in the same folder with workload.json file, and replace the "indices" and "corpora" fields as below (copied from https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/ae11cf74873bff91c7299257f1fbcb3cdde47987/nyc_taxis/workload.json).

  "indices": [
    {
      "name": "nyc_taxis",
      "body": "index.json"
    }
  ],
  "corpora": [
    {
      "name": "nyc_taxis",
      "base-url": "https://opensearch-benchmark-workloads.s3.amazonaws.com/corpora/nyc_taxis",
      "documents": [
        {
          "source-file": "documents.json.bz2",
          "document-count": 165346692,
          "compressed-bytes": 4820107188,
          "uncompressed-bytes": 79802445255
        }
      ]
    }
  ],

The command to run the benchmark
The below command will build OpenSearch from 2.4 branch, and provision an OpenSearch node to run the workloads.
Assume the above files are placed in the folder ~/Documents/searchable_snapshot_workload

Base command: opensearch-benchmark execute_test
Assign workload path: --workload-path=~/Documents/searchable_snapshot_workload
Build OpenSearch from source in 2.4 branch: --pipeline=from-sources --revision="opensearch:2.4"
Use the custom provision configuration: --provision-config-path=~/Documents/searchable_snapshot_workload/provision_configs --provision-config-instance="defaults,searchable-snapshot"
Use plugin and assign plugin parameter: --opensearch-plugins="repository-s3" --plugin-params="~/Documents/searchable_snapshot_workload/plugin_params.json"
Assign workload parameter: --workload-params="~/Documents/searchable_snapshot_workload/workload_params.json"
Exclude the task designed for regular local index: --exclude-tasks="type:force-merge", or exclude the tasks designed for the new type of index that backed by snapshot: --exclude-tasks="type:create-snapshot-repository,type:create-snapshot,type:wait-for-snapshot-create,delete-local-index-to-create-from-snapshot,type:restore-snapshot"

tlfeng · 2022-11-03T03:40:59Z

The plan for adding first class support for testing searchable snapshot workloads to opensearch-benchmark

The only action - Add a "workload" to opensearch-benchmark-workloads repository
A "workload" describes one or more benchmarking scenarios. The new workload is used for measuring the performance of Searchable Snapshots feature.
The new folder that stores the new workload will be added to the base directory of the official workload repository https://github.com/opensearch-project/opensearch-benchmark-workloads

The detailed benchmarking scenario has not been decided yet, but it can be copied from the existing workloads in the above repository for a convenient comparsion.

While the scenario can vary, the following operations is the core procedures for Searchable Snapshots feature, and need to be included.
Since opensearch-benchmark workloads can use the Jinja templating language, the following core block can also be reused by using macro syntax.

    {
      "operation": {
        "operation-type": "create-snapshot-repository",
        "repository": "test-repository",
        "body": {
          "type": "s3",
          "settings": {
            "bucket": "{{ s3_bucket_name }}",
            "region": "{{ s3_bucket_region }}"
          }
        }
      }
    },
    {
      "operation": {
        "operation-type": "create-snapshot",
        "repository": "{{ repository_name | default('test-repository') }}",
        "snapshot": "{{ snapshot_name | default('test-snapshot') }}",
        "body": {
          "indices": []
        }
      }
    },
    {
      "operation": {
        "operation-type": "wait-for-snapshot-create",
        "repository": "{{ repository_name | default('test-repository') }}",
        "snapshot": "{{ snapshot_name | default('test-snapshot') }}"
      }
    },
    {
      "operation": {
        "name": "delete-local-index-to-create-from-snapshot",
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "restore-snapshot",
        "repository": "{{ repository_name | default('test-repository') }}",
        "snapshot": "{{ snapshot_name | default('test-snapshot') }}",
        "body": {
          "indices": [],
          "storage_type": "remote_snapshot"
        },
        "wait-for-completion": true,
        "request-params": {
          "request_timeout": 3600
        }
      }
    },

Local configuration
There are additional configuration needed for running the workload with Searchable Snapshots feature enabled, but the configurations can be made locally, without uploading to the official workload repository.

Configure the remote storage -- AWS account setting: set an Amazon s3 bucket with proper permission, and set an IAM user to get AWS access key and secret key. (Reference: https://dev.to/mhihasan/elasticsearch-snapshot-and-restore-with-aws-s3-1cbp , https://opensearch.org/docs/2.3/opensearch/snapshots/snapshot-restore#amazon-s3)
Parameters for the plugin repository-s3
Create file plugin_params.json:

{
  "s3_client_name": "default",
  "s3_access_key": "<your AWS access key>",
  "s3_secret_key": "<your AWS secret key>",
}

Note: The file name can be changed, but the three fields are mandatory.

Parameters for the workload.
Create file workload_parmas.json:

{
  "snapshot_name": "<your snapshot name>"
}

Note: The file name can be changed, the "snapshot_name" field is optional, but it's useful when running the workloads multiple times. There is no operation defined by opensearch-benchmark to delete a snapshot, so different snapshot name is needed to run multiple times.

Parameter for the "provision_config_instance".
A "provision_config_instance" is a specific configuration of OpenSearch. The parameter are used for setting feature flag in jvm options to enable experimental features, and set the node role search.
Note that built-in instances can be seen from the directory https://github.com/opensearch-project/opensearch-benchmark/tree/0.1.0/osbenchmark/resources/provision_configs/main/provision_config_instances/v1, and the parameters usage can be seen here https://github.com/opensearch-project/opensearch-benchmark/blob/0.1.0/osbenchmark/resources/provision_configs/main/provision_config_instances/v1/vanilla/README.md .

Create file provision_config_instance_parmas.json:

{
  "additional_cluster_settings": {
    "node.roles": "ingest, remote_cluster_client, data, cluster_manager, search"
  },
  "additional_java_settings": [
    "-Dopensearch.experimental.feature.searchable_snapshot.enabled=true"
  ]
}

Note: The file name can be changed.

Run the performance test / benchmark
Then the command to run tests based on the workload is like:
opensearch-benchmark execute_test --distribution-version=<opensearch version> --workload=<the new workload name>

Additional required parameter for this workload:
--opensearch-plugins="repository-s3" (install the plugin when provisioning the cluster)
--provision-config-instance-params="/path/to/provision_config_instance_params.json"
--plugin-params=/path/to/plugin_params.json"
--workload-params="/path/to/workload_params.json"

Some useful optional parameter:
--workload-path=path/to/workload_json_file (when using a local workload)
--provision-config-instance="16gheap (when changing the jvm heap size, whose default value is 1GB)
--pipeline=from-sources --revision=<branch or tag name, or hash value for a commit> (when want to use an OpenSearch server that built from source)
--exclude-tasks="type:force-merge,wait-until-merges-finish" (when skipping some steps in a workload)
--test-procedure=<test scenario> (when running a specific test scenario. This parameter is valid when running the official "nyc_taxis" workload, because it has different scenarios, whose default value is "append-no-conflicts")
--client-options="{\"default\":{\"timeout\": 1200}}" (to extend the client timeout value, whose default value is 60 seconds)

[Update on 11/13/2022]
The task will be tracked in issue #5218

tlfeng · 2022-11-04T06:19:15Z

The next steps to fully accomplish the goal in the issue:

Post initial benchmarking result here.
I'm going to modify "nyc_taxis" workload with its default scenario "append-no-conflicts" to run the performance test for Searchable Snapshot feature, because currently this workload is run nightly for OpenSearch (see the code).

The modification plan is to add the "core block" of creating and restoring snapshot that mentioned in my above comment, to the code here, between ingesting data and perform search requests of the original workload.
(The "force merge" step can be excluded during the test run by adding --exclude-tasks="type:force-merge,wait-until-merges-finish" into the command, but it's not necessary because the snapshot is restored after the "force merge".)

Then use the command to run the benchmark: opensearch-benchmark execute_test --workload-path=~/Documents/searchable_snapshot_workload --pipeline=from-sources --revision=2.4 --provision-config-instance="16gheap" --provision-config-instance-params="~/Documents/searchable_snapshot_workload/provision_config_instance_params.json" --opensearch-plugins="repository-s3" --plugin-params="~/Documents/searchable_snapshot_workload/plugin_params.json" --workload-params="~/Documents/searchable_snapshot_workload/workload_params.json" --exclude-tasks="type:force-merge,wait-until-merges-finish" --test-procedure=append-no-conflicts

Decide the detailed "workload" (benchmarking scenario) to be added into the official workload repository, and create an issue there, and then following with a PR for the code change.
The workload describe in above lines is an option.
While there is a discuss about improving the performance test, [Discuss] Performance benchmarking improvements for Opensearch #3983, it reflects that there is no first class support for any advanced features in the existing workload, so I don't think the step of adding new workload for searchable snapshot can proceed alone easily.

tlfeng · 2022-11-04T17:13:04Z

Post a benchmark result here, when using nyc_taxis workload (https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/nyc_taxis), with the default scenario append-no-conflicts, and running with parameter --test-mode:

[Update on 11/13/2022]
For benchmark result based on full "nyc_taxis" workload (without parameter --test-mode), please see a following comment #4797 (comment)

In test mode:

Iteration-based tasks run at most one warmup iteration and one measurement iteration.
Time-period-based tasks run at most for 10 seconds without warmup.
Index data is postprocessed to contain 1000 documents.

The contender is a modified workload using searchable snapshot feature, with an index backed with remote snapshot.
The baseline is the original workload, using without using snapshot, with a local index.
(The workload modification to support searchable snapshot feature is adding the "core procedures" of snapshot operations that mentioned in #4797 (comment) to the original "nyc_taxis" workload)

Running in an EC2 instance, type c5.18xlarge, Ubuntu 20.04 .

opensearch-benchmark compare --baseline=86b1970a-32c9-4a4e-9d0a-80eb899501b8 --contender=3fc6b118-2458-48d4-8ade-fb14774cac5e

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/


Comparing baseline
  TestExecution ID: 86b1970a-32c9-4a4e-9d0a-80eb899501b8
  TestExecution timestamp: 2022-11-04 17:06:57
  TestProcedure: append-no-conflicts
  ProvisionConfigInstance: 16gheap

with contender
  TestExecution ID: 3fc6b118-2458-48d4-8ade-fb14774cac5e
  TestExecution timestamp: 2022-11-04 17:01:13
  TestProcedure: append-no-conflicts
  ProvisionConfigInstance: 16gheap+searchable-snapshot

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                        Metric |                Task |    Baseline |   Contender |     Diff |   Unit |
|--------------------------------------------------------------:|--------------------:|------------:|------------:|---------:|-------:|
|                    Cumulative indexing time of primary shards |                     |   0.0156833 |           0 | -0.01568 |    min |
|             Min cumulative indexing time across primary shard |                     |   0.0156833 |           0 | -0.01568 |    min |
|          Median cumulative indexing time across primary shard |                     |   0.0156833 |           0 | -0.01568 |    min |
|             Max cumulative indexing time across primary shard |                     |   0.0156833 |           0 | -0.01568 |    min |
|           Cumulative indexing throttle time of primary shards |                     |           0 |           0 |        0 |    min |
|    Min cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
| Median cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|    Max cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|                       Cumulative merge time of primary shards |                     |           0 |           0 |        0 |    min |
|                      Cumulative merge count of primary shards |                     |           0 |           0 |        0 |        |
|                Min cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|             Median cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|                Max cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|              Cumulative merge throttle time of primary shards |                     |           0 |           0 |        0 |    min |
|       Min cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|    Median cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|       Max cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|                     Cumulative refresh time of primary shards |                     |  0.00223333 |           0 | -0.00223 |    min |
|                    Cumulative refresh count of primary shards |                     |           5 |           0 |       -5 |        |
|              Min cumulative refresh time across primary shard |                     |  0.00223333 |           0 | -0.00223 |    min |
|           Median cumulative refresh time across primary shard |                     |  0.00223333 |           0 | -0.00223 |    min |
|              Max cumulative refresh time across primary shard |                     |  0.00223333 |           0 | -0.00223 |    min |
|                       Cumulative flush time of primary shards |                     |           0 |           0 |        0 |    min |
|                      Cumulative flush count of primary shards |                     |           0 |           0 |        0 |        |
|                Min cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|             Median cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|                Max cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|                                       Total Young Gen GC time |                     |           0 |           0 |        0 |      s |
|                                      Total Young Gen GC count |                     |           0 |           0 |        0 |        |
|                                         Total Old Gen GC time |                     |           0 |           0 |        0 |      s |
|                                        Total Old Gen GC count |                     |           0 |           0 |        0 |        |
|                                                    Store size |                     | 0.000252362 | 0.000254965 |        0 |     GB |
|                                                 Translog size |                     | 5.12227e-08 |           0 |       -0 |     GB |
|                                        Heap used for segments |                     |           0 |           0 |        0 |     MB |
|                                      Heap used for doc values |                     |           0 |           0 |        0 |     MB |
|                                           Heap used for terms |                     |           0 |           0 |        0 |     MB |
|                                           Heap used for norms |                     |           0 |           0 |        0 |     MB |
|                                          Heap used for points |                     |           0 |           0 |        0 |     MB |
|                                   Heap used for stored fields |                     |           0 |           0 |        0 |     MB |
|                                                 Segment count |                     |           8 |           8 |        0 |        |
|                                                Min Throughput |               index |     4498.75 |     4402.39 | -96.3607 | docs/s |
|                                               Mean Throughput |               index |     4498.75 |     4402.39 | -96.3607 | docs/s |
|                                             Median Throughput |               index |     4498.75 |     4402.39 | -96.3607 | docs/s |
|                                                Max Throughput |               index |     4498.75 |     4402.39 | -96.3607 | docs/s |
|                                       50th percentile latency |               index |     204.216 |     211.994 |  7.77821 |     ms |
|                                      100th percentile latency |               index |     213.591 |     218.374 |   4.7832 |     ms |
|                                  50th percentile service time |               index |     204.216 |     211.994 |  7.77821 |     ms |
|                                 100th percentile service time |               index |     213.591 |     218.374 |   4.7832 |     ms |
|                                                    error rate |               index |           0 |           0 |        0 |      % |
|                                                Min Throughput |             default |     17.0141 |     10.0024 | -7.01168 |  ops/s |
|                                               Mean Throughput |             default |     17.0141 |     10.0024 | -7.01168 |  ops/s |
|                                             Median Throughput |             default |     17.0141 |     10.0024 | -7.01168 |  ops/s |
|                                                Max Throughput |             default |     17.0141 |     10.0024 | -7.01168 |  ops/s |
|                                      100th percentile latency |             default |      64.695 |     134.064 |  69.3691 |     ms |
|                                 100th percentile service time |             default |     5.61543 |     33.7547 |  28.1393 |     ms |
|                                                    error rate |             default |           0 |           0 |        0 |      % |
|                                                Min Throughput |               range |      53.153 |     2.06244 | -51.0906 |  ops/s |
|                                               Mean Throughput |               range |      53.153 |     2.06244 | -51.0906 |  ops/s |
|                                             Median Throughput |               range |      53.153 |     2.06244 | -51.0906 |  ops/s |
|                                                Max Throughput |               range |      53.153 |     2.06244 | -51.0906 |  ops/s |
|                                      100th percentile latency |               range |     24.9561 |     914.374 |  889.418 |     ms |
|                                 100th percentile service time |               range |     5.86243 |     429.044 |  423.182 |     ms |
|                                                    error rate |               range |           0 |           0 |        0 |      % |
|                                                Min Throughput | distance_amount_agg |     29.9656 |     25.7695 | -4.19615 |  ops/s |
|                                               Mean Throughput | distance_amount_agg |     29.9656 |     25.7695 | -4.19615 |  ops/s |
|                                             Median Throughput | distance_amount_agg |     29.9656 |     25.7695 | -4.19615 |  ops/s |
|                                                Max Throughput | distance_amount_agg |     29.9656 |     25.7695 | -4.19615 |  ops/s |
|                                      100th percentile latency | distance_amount_agg |     38.4376 |     44.4476 |  6.01003 |     ms |
|                                 100th percentile service time | distance_amount_agg |     4.77297 |     5.28203 |  0.50905 |     ms |
|                                                    error rate | distance_amount_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |       autohisto_agg |     45.1047 |     5.93174 | -39.1729 |  ops/s |
|                                               Mean Throughput |       autohisto_agg |     45.1047 |     5.93174 | -39.1729 |  ops/s |
|                                             Median Throughput |       autohisto_agg |     45.1047 |     5.93174 | -39.1729 |  ops/s |
|                                                Max Throughput |       autohisto_agg |     45.1047 |     5.93174 | -39.1729 |  ops/s |
|                                      100th percentile latency |       autohisto_agg |     28.1104 |      295.84 |   267.73 |     ms |
|                                 100th percentile service time |       autohisto_agg |     5.66798 |     126.905 |  121.237 |     ms |
|                                                    error rate |       autohisto_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |  date_histogram_agg |     87.7203 |     7.12393 | -80.5964 |  ops/s |
|                                               Mean Throughput |  date_histogram_agg |     87.7203 |     7.12393 | -80.5964 |  ops/s |
|                                             Median Throughput |  date_histogram_agg |     87.7203 |     7.12393 | -80.5964 |  ops/s |
|                                                Max Throughput |  date_histogram_agg |     87.7203 |     7.12393 | -80.5964 |  ops/s |
|                                      100th percentile latency |  date_histogram_agg |     16.8181 |     274.741 |  257.922 |     ms |
|                                 100th percentile service time |  date_histogram_agg |     5.12861 |     133.956 |  128.827 |     ms |
|                                                    error rate |  date_histogram_agg |           0 |           0 |        0 |      % |

The below are separate results:

With Searchable Snapshots:
--exclude-tasks="type:force-merge,wait-until-merges-finish" --test-mode

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] Preparing for test execution ...
[INFO] Executing test with workload [nyc_taxis], test_procedure [append-no-conflicts] and provision_config_instance ['16gheap', 'searchable-snapshot'] with version [2.4.0-SNAPSHOT].

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index                                                                  [100% done]
Running refresh-after-index                                                    [100% done]
Running refresh-after-force-merge                                              [100% done]
Running create-snapshot-repository                                             [100% done]
Running create-snapshot                                                        [100% done]
Running wait-for-snapshot-create                                               [100% done]
Running delete-local-index-to-create-from-snapshot                             [100% done]
Running restore-snapshot                                                       [100% done]
Running default                                                                [100% done]
Running range                                                                  [100% done]
Running distance_amount_agg                                                    [100% done]
Running autohisto_agg                                                          [100% done]
Running date_histogram_agg                                                     [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |           0 |    min |
|             Min cumulative indexing time across primary shards |                          |           0 |    min |
|          Median cumulative indexing time across primary shards |                          |           0 |    min |
|             Max cumulative indexing time across primary shards |                          |           0 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |           0 |    min |
|                       Cumulative merge count of primary shards |                          |           0 |        |
|                Min cumulative merge time across primary shards |                          |           0 |    min |
|             Median cumulative merge time across primary shards |                          |           0 |    min |
|                Max cumulative merge time across primary shards |                          |           0 |    min |
|               Cumulative merge throttle time of primary shards |                          |           0 |    min |
|       Min cumulative merge throttle time across primary shards |                          |           0 |    min |
|    Median cumulative merge throttle time across primary shards |                          |           0 |    min |
|       Max cumulative merge throttle time across primary shards |                          |           0 |    min |
|                      Cumulative refresh time of primary shards |                          |           0 |    min |
|                     Cumulative refresh count of primary shards |                          |           0 |        |
|              Min cumulative refresh time across primary shards |                          |           0 |    min |
|           Median cumulative refresh time across primary shards |                          |           0 |    min |
|              Max cumulative refresh time across primary shards |                          |           0 |    min |
|                        Cumulative flush time of primary shards |                          |           0 |    min |
|                       Cumulative flush count of primary shards |                          |           0 |        |
|                Min cumulative flush time across primary shards |                          |           0 |    min |
|             Median cumulative flush time across primary shards |                          |           0 |    min |
|                Max cumulative flush time across primary shards |                          |           0 |    min |
|                                        Total Young Gen GC time |                          |           0 |      s |
|                                       Total Young Gen GC count |                          |           0 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          | 0.000254965 |     GB |
|                                                  Translog size |                          |           0 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |           8 |        |
|                                                 Min Throughput |                    index |     4402.39 | docs/s |
|                                                Mean Throughput |                    index |     4402.39 | docs/s |
|                                              Median Throughput |                    index |     4402.39 | docs/s |
|                                                 Max Throughput |                    index |     4402.39 | docs/s |
|                                        50th percentile latency |                    index |     211.994 |     ms |
|                                       100th percentile latency |                    index |     218.374 |     ms |
|                                   50th percentile service time |                    index |     211.994 |     ms |
|                                  100th percentile service time |                    index |     218.374 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-for-snapshot-create |      684418 | byte/s |
|                                                Mean Throughput | wait-for-snapshot-create |      684418 | byte/s |
|                                              Median Throughput | wait-for-snapshot-create |      684418 | byte/s |
|                                                 Max Throughput | wait-for-snapshot-create |      684418 | byte/s |
|                                       100th percentile latency | wait-for-snapshot-create |     1255.97 |     ms |
|                                  100th percentile service time | wait-for-snapshot-create |     1255.97 |     ms |
|                                                     error rate | wait-for-snapshot-create |           0 |      % |
|                                                 Min Throughput |                  default |          10 |  ops/s |
|                                                Mean Throughput |                  default |          10 |  ops/s |
|                                              Median Throughput |                  default |          10 |  ops/s |
|                                                 Max Throughput |                  default |          10 |  ops/s |
|                                       100th percentile latency |                  default |     134.064 |     ms |
|                                  100th percentile service time |                  default |     33.7547 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |        2.06 |  ops/s |
|                                                Mean Throughput |                    range |        2.06 |  ops/s |
|                                              Median Throughput |                    range |        2.06 |  ops/s |
|                                                 Max Throughput |                    range |        2.06 |  ops/s |
|                                       100th percentile latency |                    range |     914.374 |     ms |
|                                  100th percentile service time |                    range |     429.044 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |       25.77 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |       25.77 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |       25.77 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |       25.77 |  ops/s |
|                                       100th percentile latency |      distance_amount_agg |     44.4476 |     ms |
|                                  100th percentile service time |      distance_amount_agg |     5.28203 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |        5.93 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |        5.93 |  ops/s |
|                                              Median Throughput |            autohisto_agg |        5.93 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |        5.93 |  ops/s |
|                                       100th percentile latency |            autohisto_agg |      295.84 |     ms |
|                                  100th percentile service time |            autohisto_agg |     126.905 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |        7.12 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |        7.12 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |        7.12 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        7.12 |  ops/s |
|                                       100th percentile latency |       date_histogram_agg |     274.741 |     ms |
|                                  100th percentile service time |       date_histogram_agg |     133.956 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


--------------------------------
[INFO] SUCCESS (took 36 seconds)
--------------------------------

Without using Searchable Snapshots:
--exclude-tasks="type:create-snapshot-repository,type:create-snapshot,type:wait-for-snapshot-create,delete-local-index-to-create-from-snapshot,type:restore-snapshot" --test-mode

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] Preparing for test execution ...
[INFO] Executing test with workload [nyc_taxis], test_procedure [append-no-conflicts] and provision_config_instance ['16gheap'] with version [2.4.0-SNAPSHOT].

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index                                                                  [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running default                                                                [100% done]
Running range                                                                  [100% done]
Running distance_amount_agg                                                    [100% done]
Running autohisto_agg                                                          [100% done]
Running date_histogram_agg                                                     [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |   0.0156833 |    min |
|             Min cumulative indexing time across primary shards |                          |   0.0156833 |    min |
|          Median cumulative indexing time across primary shards |                          |   0.0156833 |    min |
|             Max cumulative indexing time across primary shards |                          |   0.0156833 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |           0 |    min |
|                       Cumulative merge count of primary shards |                          |           0 |        |
|                Min cumulative merge time across primary shards |                          |           0 |    min |
|             Median cumulative merge time across primary shards |                          |           0 |    min |
|                Max cumulative merge time across primary shards |                          |           0 |    min |
|               Cumulative merge throttle time of primary shards |                          |           0 |    min |
|       Min cumulative merge throttle time across primary shards |                          |           0 |    min |
|    Median cumulative merge throttle time across primary shards |                          |           0 |    min |
|       Max cumulative merge throttle time across primary shards |                          |           0 |    min |
|                      Cumulative refresh time of primary shards |                          |  0.00223333 |    min |
|                     Cumulative refresh count of primary shards |                          |           5 |        |
|              Min cumulative refresh time across primary shards |                          |  0.00223333 |    min |
|           Median cumulative refresh time across primary shards |                          |  0.00223333 |    min |
|              Max cumulative refresh time across primary shards |                          |  0.00223333 |    min |
|                        Cumulative flush time of primary shards |                          |           0 |    min |
|                       Cumulative flush count of primary shards |                          |           0 |        |
|                Min cumulative flush time across primary shards |                          |           0 |    min |
|             Median cumulative flush time across primary shards |                          |           0 |    min |
|                Max cumulative flush time across primary shards |                          |           0 |    min |
|                                        Total Young Gen GC time |                          |           0 |      s |
|                                       Total Young Gen GC count |                          |           0 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          | 0.000252362 |     GB |
|                                                  Translog size |                          | 5.12227e-08 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |           8 |        |
|                                                 Min Throughput |                    index |     4498.75 | docs/s |
|                                                Mean Throughput |                    index |     4498.75 | docs/s |
|                                              Median Throughput |                    index |     4498.75 | docs/s |
|                                                 Max Throughput |                    index |     4498.75 | docs/s |
|                                        50th percentile latency |                    index |     204.216 |     ms |
|                                       100th percentile latency |                    index |     213.591 |     ms |
|                                   50th percentile service time |                    index |     204.216 |     ms |
|                                  100th percentile service time |                    index |     213.591 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |       81.49 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |       81.49 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |       81.49 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |       81.49 |  ops/s |
|                                       100th percentile latency | wait-until-merges-finish |     11.7257 |     ms |
|                                  100th percentile service time | wait-until-merges-finish |     11.7257 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |       17.01 |  ops/s |
|                                                Mean Throughput |                  default |       17.01 |  ops/s |
|                                              Median Throughput |                  default |       17.01 |  ops/s |
|                                                 Max Throughput |                  default |       17.01 |  ops/s |
|                                       100th percentile latency |                  default |      64.695 |     ms |
|                                  100th percentile service time |                  default |     5.61543 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |       53.15 |  ops/s |
|                                                Mean Throughput |                    range |       53.15 |  ops/s |
|                                              Median Throughput |                    range |       53.15 |  ops/s |
|                                                 Max Throughput |                    range |       53.15 |  ops/s |
|                                       100th percentile latency |                    range |     24.9561 |     ms |
|                                  100th percentile service time |                    range |     5.86243 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |       29.97 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |       29.97 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |       29.97 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |       29.97 |  ops/s |
|                                       100th percentile latency |      distance_amount_agg |     38.4376 |     ms |
|                                  100th percentile service time |      distance_amount_agg |     4.77297 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |        45.1 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |        45.1 |  ops/s |
|                                              Median Throughput |            autohisto_agg |        45.1 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |        45.1 |  ops/s |
|                                       100th percentile latency |            autohisto_agg |     28.1104 |     ms |
|                                  100th percentile service time |            autohisto_agg |     5.66798 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |       87.72 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |       87.72 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |       87.72 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |       87.72 |  ops/s |
|                                       100th percentile latency |       date_histogram_agg |     16.8181 |     ms |
|                                  100th percentile service time |       date_histogram_agg |     5.12861 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


--------------------------------
[INFO] SUCCESS (took 29 seconds)
--------------------------------

tlfeng · 2022-11-04T19:05:35Z

Close this issue since the goad has been achieved.
Initial benchmark results:
1 - first 1k documents of "nyc_taxis" workload #4797 (comment)
2 - full "nyc_taxis" workload #4797 (comment)
Plan to support running benchmark for the feature directly: #4797 (comment)
More explanation about the initial result and the further plan: #4797 (comment)

tlfeng · 2022-11-10T09:02:19Z

A benchmark result using the complete nyc_taxis workload (https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/nyc_taxis)
The contender is a modified workload using searchable snapshot feature, with an index backed with remote snapshot.
The baseline is the original workload, using without using snapshot, with a local index.
(The workload modification to support searchable snapshot feature is adding the "core procedures" of snapshot operations that mentioned in #4797 (comment) to the original "nyc_taxis" workload)

Running in an EC2 instance, type c5.18xlarge, Ubuntu 20.04 .

opensearch-benchmark compare --baseline=41252cea-6757-4a98-870b-5def0f428a00 --contender=faf47c9a-822c-408a-885c-0cee8a32ffc2


   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/


Comparing baseline
  TestExecution ID: 41252cea-6757-4a98-870b-5def0f428a00
  TestExecution timestamp: 2022-11-10 07:54:46
  TestProcedure: append-no-conflicts
  ProvisionConfigInstance: 24gheap

with contender
  TestExecution ID: faf47c9a-822c-408a-885c-0cee8a32ffc2
  TestExecution timestamp: 2022-11-10 09:11:50
  TestProcedure: append-no-conflicts
  ProvisionConfigInstance: 24gheap

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                        Metric |                Task |    Baseline |   Contender |        Diff |   Unit |
|--------------------------------------------------------------:|--------------------:|------------:|------------:|------------:|-------:|
|                    Cumulative indexing time of primary shards |                     |     145.177 |           0 |    -145.177 |    min |
|             Min cumulative indexing time across primary shard |                     |     145.177 |           0 |    -145.177 |    min |
|          Median cumulative indexing time across primary shard |                     |     145.177 |           0 |    -145.177 |    min |
|             Max cumulative indexing time across primary shard |                     |     145.177 |           0 |    -145.177 |    min |
|           Cumulative indexing throttle time of primary shards |                     |           0 |           0 |           0 |    min |
|    Min cumulative indexing throttle time across primary shard |                     |           0 |           0 |           0 |    min |
| Median cumulative indexing throttle time across primary shard |                     |           0 |           0 |           0 |    min |
|    Max cumulative indexing throttle time across primary shard |                     |           0 |           0 |           0 |    min |
|                       Cumulative merge time of primary shards |                     |     42.8124 |           0 |    -42.8124 |    min |
|                      Cumulative merge count of primary shards |                     |          87 |           0 |         -87 |        |
|                Min cumulative merge time across primary shard |                     |     42.8124 |           0 |    -42.8124 |    min |
|             Median cumulative merge time across primary shard |                     |     42.8124 |           0 |    -42.8124 |    min |
|                Max cumulative merge time across primary shard |                     |     42.8124 |           0 |    -42.8124 |    min |
|              Cumulative merge throttle time of primary shards |                     |     8.04287 |           0 |    -8.04287 |    min |
|       Min cumulative merge throttle time across primary shard |                     |     8.04287 |           0 |    -8.04287 |    min |
|    Median cumulative merge throttle time across primary shard |                     |     8.04287 |           0 |    -8.04287 |    min |
|       Max cumulative merge throttle time across primary shard |                     |     8.04287 |           0 |    -8.04287 |    min |
|                     Cumulative refresh time of primary shards |                     |     2.66532 |           0 |    -2.66532 |    min |
|                    Cumulative refresh count of primary shards |                     |          80 |           0 |         -80 |        |
|              Min cumulative refresh time across primary shard |                     |     2.66532 |           0 |    -2.66532 |    min |
|           Median cumulative refresh time across primary shard |                     |     2.66532 |           0 |    -2.66532 |    min |
|              Max cumulative refresh time across primary shard |                     |     2.66532 |           0 |    -2.66532 |    min |
|                       Cumulative flush time of primary shards |                     |     2.87093 |           0 |    -2.87093 |    min |
|                      Cumulative flush count of primary shards |                     |          34 |           1 |         -33 |        |
|                Min cumulative flush time across primary shard |                     |     2.87093 |           0 |    -2.87093 |    min |
|             Median cumulative flush time across primary shard |                     |     2.87093 |           0 |    -2.87093 |    min |
|                Max cumulative flush time across primary shard |                     |     2.87093 |           0 |    -2.87093 |    min |
|                                       Total Young Gen GC time |                     |      14.073 |      15.284 |       1.211 |      s |
|                                      Total Young Gen GC count |                     |         341 |         434 |          93 |        |
|                                         Total Old Gen GC time |                     |           0 |           0 |           0 |      s |
|                                        Total Old Gen GC count |                     |           0 |           0 |           0 |        |
|                                                    Store size |                     |     24.5266 |     24.3569 |    -0.16969 |     GB |
|                                                 Translog size |                     | 5.12227e-08 |           0 |          -0 |     GB |
|                                        Heap used for segments |                     |           0 |           0 |           0 |     MB |
|                                      Heap used for doc values |                     |           0 |           0 |           0 |     MB |
|                                           Heap used for terms |                     |           0 |           0 |           0 |     MB |
|                                           Heap used for norms |                     |           0 |           0 |           0 |     MB |
|                                          Heap used for points |                     |           0 |           0 |           0 |     MB |
|                                   Heap used for stored fields |                     |           0 |           0 |           0 |     MB |
|                                                 Segment count |                     |          35 |          47 |          12 |        |
|                                                Min Throughput |               index |      121400 |      123739 |     2338.72 | docs/s |
|                                               Mean Throughput |               index |      127229 |      127962 |     732.711 | docs/s |
|                                             Median Throughput |               index |      127967 |      128415 |     447.569 | docs/s |
|                                                Max Throughput |               index |      129506 |      129559 |     53.2926 | docs/s |
|                                       50th percentile latency |               index |     523.462 |     522.601 |    -0.86098 |     ms |
|                                       90th percentile latency |               index |     639.903 |     634.403 |     -5.5009 |     ms |
|                                       99th percentile latency |               index |     2661.57 |     3010.43 |     348.856 |     ms |
|                                     99.9th percentile latency |               index |     4973.25 |     5103.78 |     130.523 |     ms |
|                                    99.99th percentile latency |               index |     7250.47 |     7909.71 |     659.232 |     ms |
|                                      100th percentile latency |               index |     8385.43 |     8704.86 |     319.429 |     ms |
|                                  50th percentile service time |               index |     523.462 |     522.601 |    -0.86098 |     ms |
|                                  90th percentile service time |               index |     639.903 |     634.403 |     -5.5009 |     ms |
|                                  99th percentile service time |               index |     2661.57 |     3010.43 |     348.856 |     ms |
|                                99.9th percentile service time |               index |     4973.25 |     5103.78 |     130.523 |     ms |
|                               99.99th percentile service time |               index |     7250.47 |     7909.71 |     659.232 |     ms |
|                                 100th percentile service time |               index |     8385.43 |     8704.86 |     319.429 |     ms |
|                                                    error rate |               index |           0 |           0 |           0 |      % |
|                                                Min Throughput |             default |     3.01568 |     2.98715 |    -0.02852 |  ops/s |
|                                               Mean Throughput |             default |     3.02553 |     2.99291 |    -0.03262 |  ops/s |
|                                             Median Throughput |             default |     3.02324 |     2.99357 |    -0.02967 |  ops/s |
|                                                Max Throughput |             default |     3.04509 |     2.99562 |    -0.04948 |  ops/s |
|                                       50th percentile latency |             default |     5.75124 |       173.7 |     167.949 |     ms |
|                                       90th percentile latency |             default |     6.39432 |     207.338 |     200.944 |     ms |
|                                       99th percentile latency |             default |     6.96071 |     255.386 |     248.425 |     ms |
|                                      100th percentile latency |             default |     8.10929 |     270.333 |     262.223 |     ms |
|                                  50th percentile service time |             default |     4.32381 |     171.866 |     167.543 |     ms |
|                                  90th percentile service time |             default |     4.97152 |      205.81 |     200.839 |     ms |
|                                  99th percentile service time |             default |     5.57359 |     252.939 |     247.366 |     ms |
|                                 100th percentile service time |             default |      7.0567 |      269.73 |     262.673 |     ms |
|                                                    error rate |             default |           0 |           0 |           0 |      % |
|                                                Min Throughput |               range |    0.703405 |  0.00228472 |    -0.70112 |  ops/s |
|                                               Mean Throughput |               range |    0.705601 |  0.00230452 |     -0.7033 |  ops/s |
|                                             Median Throughput |               range |    0.705097 |  0.00229157 |    -0.70281 |  ops/s |
|                                                Max Throughput |               range |    0.710114 |   0.0023502 |    -0.70776 |  ops/s |
|                                       50th percentile latency |               range |     247.518 | 4.37725e+07 | 4.37722e+07 |     ms |
|                                       90th percentile latency |               range |     249.543 | 6.10926e+07 | 6.10924e+07 |     ms |
|                                       99th percentile latency |               range |     257.074 | 6.47726e+07 | 6.47723e+07 |     ms |
|                                      100th percentile latency |               range |     257.644 | 6.51695e+07 | 6.51693e+07 |     ms |
|                                  50th percentile service time |               range |     245.433 |      441809 |      441564 |     ms |
|                                  90th percentile service time |               range |     246.996 |      466151 |      465904 |     ms |
|                                  99th percentile service time |               range |     254.822 |      485194 |      484940 |     ms |
|                                 100th percentile service time |               range |     255.836 |      488356 |      488101 |     ms |
|                                                    error rate |               range |           0 |           0 |           0 |      % |
|                                                Min Throughput | distance_amount_agg |     2.01213 |     2.01143 |    -0.00071 |  ops/s |
|                                               Mean Throughput | distance_amount_agg |     2.01997 |     2.01877 |     -0.0012 |  ops/s |
|                                             Median Throughput | distance_amount_agg |     2.01815 |     2.01705 |     -0.0011 |  ops/s |
|                                                Max Throughput | distance_amount_agg |     2.03591 |     2.03369 |    -0.00222 |  ops/s |
|                                       50th percentile latency | distance_amount_agg |     4.09597 |     4.70118 |     0.60521 |     ms |
|                                       90th percentile latency | distance_amount_agg |     4.68576 |     5.24222 |     0.55645 |     ms |
|                                       99th percentile latency | distance_amount_agg |     5.26528 |     5.55888 |     0.29359 |     ms |
|                                      100th percentile latency | distance_amount_agg |     5.57506 |     5.67187 |      0.0968 |     ms |
|                                  50th percentile service time | distance_amount_agg |      2.7258 |     3.11669 |     0.39089 |     ms |
|                                  90th percentile service time | distance_amount_agg |     2.98549 |     3.45944 |     0.47394 |     ms |
|                                  99th percentile service time | distance_amount_agg |     3.42588 |     3.68248 |     0.25659 |     ms |
|                                 100th percentile service time | distance_amount_agg |     4.04084 |     3.77821 |    -0.26263 |     ms |
|                                                    error rate | distance_amount_agg |           0 |           0 |           0 |      % |
|                                                Min Throughput |       autohisto_agg |     1.49882 |  0.00138305 |    -1.49743 |  ops/s |
|                                               Mean Throughput |       autohisto_agg |     1.49933 |  0.00140622 |    -1.49793 |  ops/s |
|                                             Median Throughput |       autohisto_agg |      1.4994 |  0.00140819 |    -1.49799 |  ops/s |
|                                                Max Throughput |       autohisto_agg |      1.4996 |  0.00142313 |    -1.49818 |  ops/s |
|                                       50th percentile latency |       autohisto_agg |     550.008 | 7.11091e+07 | 7.11085e+07 |     ms |
|                                       90th percentile latency |       autohisto_agg |     557.414 | 9.82427e+07 | 9.82422e+07 |     ms |
|                                       99th percentile latency |       autohisto_agg |     562.471 |  1.0455e+08 |  1.0455e+08 |     ms |
|                                      100th percentile latency |       autohisto_agg |     565.506 | 1.05277e+08 | 1.05276e+08 |     ms |
|                                  50th percentile service time |       autohisto_agg |     548.899 |      691756 |      691207 |     ms |
|                                  90th percentile service time |       autohisto_agg |      556.13 |      726229 |      725673 |     ms |
|                                  99th percentile service time |       autohisto_agg |     561.028 |      751422 |      750861 |     ms |
|                                 100th percentile service time |       autohisto_agg |     564.407 |      765487 |      764923 |     ms |
|                                                    error rate |       autohisto_agg |           0 |           0 |           0 |      % |
|                                                Min Throughput |  date_histogram_agg |     1.50185 |  0.00148547 |    -1.50037 |  ops/s |
|                                               Mean Throughput |  date_histogram_agg |     1.50303 |  0.00149806 |    -1.50153 |  ops/s |
|                                             Median Throughput |  date_histogram_agg |     1.50276 |  0.00149942 |    -1.50126 |  ops/s |
|                                                Max Throughput |  date_histogram_agg |     1.50539 |  0.00150452 |    -1.50388 |  ops/s |
|                                       50th percentile latency |  date_histogram_agg |     525.286 |    6.68e+07 | 6.67995e+07 |     ms |
|                                       90th percentile latency |  date_histogram_agg |     530.071 | 9.33575e+07 | 9.33569e+07 |     ms |
|                                       99th percentile latency |  date_histogram_agg |     537.064 | 9.92378e+07 | 9.92372e+07 |     ms |
|                                      100th percentile latency |  date_histogram_agg |       539.4 | 9.98723e+07 | 9.98718e+07 |     ms |
|                                  50th percentile service time |  date_histogram_agg |     523.732 |      662967 |      662444 |     ms |
|                                  90th percentile service time |  date_histogram_agg |     528.677 |      689567 |      689038 |     ms |
|                                  99th percentile service time |  date_histogram_agg |     534.724 |      700547 |      700012 |     ms |
|                                 100th percentile service time |  date_histogram_agg |     537.719 |      714373 |      713836 |     ms |
|                                                    error rate |  date_histogram_agg |           0 |           0 |           0 |      % |

The below is the separate result for the two executions.
This one using the original "nyc_taxis" workload, using local index, as the baseline.

opensearch-benchmark execute_test --workload=nyc_taxis --pipeline=from-sources --revision="opensearch:2.4" --provision-config-instance="24gheap"

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] Preparing for test execution ...
[INFO] Executing test with workload [nyc_taxis], test_procedure [append-no-conflicts] and provision_config_instance ['24gheap'] with version [2.4.0-SNAPSHOT].

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index                                                                  [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running default                                                                [100% done]
Running range                                                                  [100% done]
Running distance_amount_agg                                                    [100% done]
Running autohisto_agg                                                          [100% done]
Running date_histogram_agg                                                     [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |     145.177 |    min |
|             Min cumulative indexing time across primary shards |                          |     145.177 |    min |
|          Median cumulative indexing time across primary shards |                          |     145.177 |    min |
|             Max cumulative indexing time across primary shards |                          |     145.177 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |     42.8124 |    min |
|                       Cumulative merge count of primary shards |                          |          87 |        |
|                Min cumulative merge time across primary shards |                          |     42.8124 |    min |
|             Median cumulative merge time across primary shards |                          |     42.8124 |    min |
|                Max cumulative merge time across primary shards |                          |     42.8124 |    min |
|               Cumulative merge throttle time of primary shards |                          |     8.04287 |    min |
|       Min cumulative merge throttle time across primary shards |                          |     8.04287 |    min |
|    Median cumulative merge throttle time across primary shards |                          |     8.04287 |    min |
|       Max cumulative merge throttle time across primary shards |                          |     8.04287 |    min |
|                      Cumulative refresh time of primary shards |                          |     2.66532 |    min |
|                     Cumulative refresh count of primary shards |                          |          80 |        |
|              Min cumulative refresh time across primary shards |                          |     2.66532 |    min |
|           Median cumulative refresh time across primary shards |                          |     2.66532 |    min |
|              Max cumulative refresh time across primary shards |                          |     2.66532 |    min |
|                        Cumulative flush time of primary shards |                          |     2.87093 |    min |
|                       Cumulative flush count of primary shards |                          |          34 |        |
|                Min cumulative flush time across primary shards |                          |     2.87093 |    min |
|             Median cumulative flush time across primary shards |                          |     2.87093 |    min |
|                Max cumulative flush time across primary shards |                          |     2.87093 |    min |
|                                        Total Young Gen GC time |                          |      14.073 |      s |
|                                       Total Young Gen GC count |                          |         341 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     24.5266 |     GB |
|                                                  Translog size |                          | 5.12227e-08 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |          35 |        |
|                                                 Min Throughput |                    index |      121400 | docs/s |
|                                                Mean Throughput |                    index |      127229 | docs/s |
|                                              Median Throughput |                    index |      127967 | docs/s |
|                                                 Max Throughput |                    index |      129506 | docs/s |
|                                        50th percentile latency |                    index |     523.462 |     ms |
|                                        90th percentile latency |                    index |     639.903 |     ms |
|                                        99th percentile latency |                    index |     2661.57 |     ms |
|                                      99.9th percentile latency |                    index |     4973.25 |     ms |
|                                     99.99th percentile latency |                    index |     7250.47 |     ms |
|                                       100th percentile latency |                    index |     8385.43 |     ms |
|                                   50th percentile service time |                    index |     523.462 |     ms |
|                                   90th percentile service time |                    index |     639.903 |     ms |
|                                   99th percentile service time |                    index |     2661.57 |     ms |
|                                 99.9th percentile service time |                    index |     4973.25 |     ms |
|                                99.99th percentile service time |                    index |     7250.47 |     ms |
|                                  100th percentile service time |                    index |     8385.43 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                                Mean Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                              Median Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                                 Max Throughput | wait-until-merges-finish |        0.01 |  ops/s |
|                                       100th percentile latency | wait-until-merges-finish |     89637.6 |     ms |
|                                  100th percentile service time | wait-until-merges-finish |     89637.6 |     ms |
|                                                     error rate | wait-until-merges-finish |           0 |      % |
|                                                 Min Throughput |                  default |        3.02 |  ops/s |
|                                                Mean Throughput |                  default |        3.03 |  ops/s |
|                                              Median Throughput |                  default |        3.02 |  ops/s |
|                                                 Max Throughput |                  default |        3.05 |  ops/s |
|                                        50th percentile latency |                  default |     5.75124 |     ms |
|                                        90th percentile latency |                  default |     6.39432 |     ms |
|                                        99th percentile latency |                  default |     6.96071 |     ms |
|                                       100th percentile latency |                  default |     8.10929 |     ms |
|                                   50th percentile service time |                  default |     4.32381 |     ms |
|                                   90th percentile service time |                  default |     4.97152 |     ms |
|                                   99th percentile service time |                  default |     5.57359 |     ms |
|                                  100th percentile service time |                  default |      7.0567 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |         0.7 |  ops/s |
|                                                Mean Throughput |                    range |        0.71 |  ops/s |
|                                              Median Throughput |                    range |        0.71 |  ops/s |
|                                                 Max Throughput |                    range |        0.71 |  ops/s |
|                                        50th percentile latency |                    range |     247.518 |     ms |
|                                        90th percentile latency |                    range |     249.543 |     ms |
|                                        99th percentile latency |                    range |     257.074 |     ms |
|                                       100th percentile latency |                    range |     257.644 |     ms |
|                                   50th percentile service time |                    range |     245.433 |     ms |
|                                   90th percentile service time |                    range |     246.996 |     ms |
|                                   99th percentile service time |                    range |     254.822 |     ms |
|                                  100th percentile service time |                    range |     255.836 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.04 |  ops/s |
|                                        50th percentile latency |      distance_amount_agg |     4.09597 |     ms |
|                                        90th percentile latency |      distance_amount_agg |     4.68576 |     ms |
|                                        99th percentile latency |      distance_amount_agg |     5.26528 |     ms |
|                                       100th percentile latency |      distance_amount_agg |     5.57506 |     ms |
|                                   50th percentile service time |      distance_amount_agg |      2.7258 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     2.98549 |     ms |
|                                   99th percentile service time |      distance_amount_agg |     3.42588 |     ms |
|                                  100th percentile service time |      distance_amount_agg |     4.04084 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                              Median Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |         1.5 |  ops/s |
|                                        50th percentile latency |            autohisto_agg |     550.008 |     ms |
|                                        90th percentile latency |            autohisto_agg |     557.414 |     ms |
|                                        99th percentile latency |            autohisto_agg |     562.471 |     ms |
|                                       100th percentile latency |            autohisto_agg |     565.506 |     ms |
|                                   50th percentile service time |            autohisto_agg |     548.899 |     ms |
|                                   90th percentile service time |            autohisto_agg |      556.13 |     ms |
|                                   99th percentile service time |            autohisto_agg |     561.028 |     ms |
|                                  100th percentile service time |            autohisto_agg |     564.407 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |         1.5 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |         1.5 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |         1.5 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |        1.51 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |     525.286 |     ms |
|                                        90th percentile latency |       date_histogram_agg |     530.071 |     ms |
|                                        99th percentile latency |       date_histogram_agg |     537.064 |     ms |
|                                       100th percentile latency |       date_histogram_agg |       539.4 |     ms |
|                                   50th percentile service time |       date_histogram_agg |     523.732 |     ms |
|                                   90th percentile service time |       date_histogram_agg |     528.677 |     ms |
|                                   99th percentile service time |       date_histogram_agg |     534.724 |     ms |
|                                  100th percentile service time |       date_histogram_agg |     537.719 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


----------------------------------
[INFO] SUCCESS (took 2004 seconds)
----------------------------------

The below is the workload with Searchable Snapshot feature:

opensearch-benchmark execute_test --workload-path=~/benchmark-workload/searchable-snapshot/nyc_taxis --pipeline=from-sources --revision=2.4 --kill-running-processes --provision-config-instance="24gheap" --provision-config-instance-params="~/benchmark-workload/searchable-snapshot/provision_config_instance_params.json" --opensearch-plugins="repository-s3" --plugin-params="~/benchmark-workload/searchable-snapshot/plugin_params.json" --workload-params="~/benchmark-workload/searchable-snapshot/workload_params.json" --exclude-tasks="type:force-merge,wait-until-merges-finish" --client-options="{\"default\":{\"timeout\": 1200}}"

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] Preparing for test execution ...
[INFO] Executing test with workload [nyc_taxis], test_procedure [append-no-conflicts] and provision_config_instance ['24gheap'] with version [2.4.0-SNAPSHOT].

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index                                                                  [100% done]
Running refresh-after-index                                                    [100% done]
Running refresh-after-force-merge                                              [100% done]
Running create-snapshot-repository                                             [100% done]
Running create-snapshot                                                        [100% done]
Running wait-for-snapshot-create                                               [100% done]
Running delete-local-index-to-create-from-snapshot                             [100% done]
Running restore-snapshot                                                       [100% done]
Running default                                                                [100% done]
Running range                                                                  [100% done]
Running distance_amount_agg                                                    [100% done]
Running autohisto_agg                                                          [100% done]
Running date_histogram_agg                                                     [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                         Metric |                     Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |                          |           0 |    min |
|             Min cumulative indexing time across primary shards |                          |           0 |    min |
|          Median cumulative indexing time across primary shards |                          |           0 |    min |
|             Max cumulative indexing time across primary shards |                          |           0 |    min |
|            Cumulative indexing throttle time of primary shards |                          |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |                          |           0 |    min |
| Median cumulative indexing throttle time across primary shards |                          |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |                          |           0 |    min |
|                        Cumulative merge time of primary shards |                          |           0 |    min |
|                       Cumulative merge count of primary shards |                          |           0 |        |
|                Min cumulative merge time across primary shards |                          |           0 |    min |
|             Median cumulative merge time across primary shards |                          |           0 |    min |
|                Max cumulative merge time across primary shards |                          |           0 |    min |
|               Cumulative merge throttle time of primary shards |                          |           0 |    min |
|       Min cumulative merge throttle time across primary shards |                          |           0 |    min |
|    Median cumulative merge throttle time across primary shards |                          |           0 |    min |
|       Max cumulative merge throttle time across primary shards |                          |           0 |    min |
|                      Cumulative refresh time of primary shards |                          |           0 |    min |
|                     Cumulative refresh count of primary shards |                          |           0 |        |
|              Min cumulative refresh time across primary shards |                          |           0 |    min |
|           Median cumulative refresh time across primary shards |                          |           0 |    min |
|              Max cumulative refresh time across primary shards |                          |           0 |    min |
|                        Cumulative flush time of primary shards |                          |           0 |    min |
|                       Cumulative flush count of primary shards |                          |           1 |        |
|                Min cumulative flush time across primary shards |                          |           0 |    min |
|             Median cumulative flush time across primary shards |                          |           0 |    min |
|                Max cumulative flush time across primary shards |                          |           0 |    min |
|                                        Total Young Gen GC time |                          |      15.284 |      s |
|                                       Total Young Gen GC count |                          |         434 |        |
|                                          Total Old Gen GC time |                          |           0 |      s |
|                                         Total Old Gen GC count |                          |           0 |        |
|                                                     Store size |                          |     24.3569 |     GB |
|                                                  Translog size |                          |           0 |     GB |
|                                         Heap used for segments |                          |           0 |     MB |
|                                       Heap used for doc values |                          |           0 |     MB |
|                                            Heap used for terms |                          |           0 |     MB |
|                                            Heap used for norms |                          |           0 |     MB |
|                                           Heap used for points |                          |           0 |     MB |
|                                    Heap used for stored fields |                          |           0 |     MB |
|                                                  Segment count |                          |          47 |        |
|                                                 Min Throughput |                    index |      123739 | docs/s |
|                                                Mean Throughput |                    index |      127962 | docs/s |
|                                              Median Throughput |                    index |      128415 | docs/s |
|                                                 Max Throughput |                    index |      129559 | docs/s |
|                                        50th percentile latency |                    index |     522.601 |     ms |
|                                        90th percentile latency |                    index |     634.403 |     ms |
|                                        99th percentile latency |                    index |     3010.43 |     ms |
|                                      99.9th percentile latency |                    index |     5103.78 |     ms |
|                                     99.99th percentile latency |                    index |     7909.71 |     ms |
|                                       100th percentile latency |                    index |     8704.86 |     ms |
|                                   50th percentile service time |                    index |     522.601 |     ms |
|                                   90th percentile service time |                    index |     634.403 |     ms |
|                                   99th percentile service time |                    index |     3010.43 |     ms |
|                                 99.9th percentile service time |                    index |     5103.78 |     ms |
|                                99.99th percentile service time |                    index |     7909.71 |     ms |
|                                  100th percentile service time |                    index |     8704.86 |     ms |
|                                                     error rate |                    index |           0 |      % |
|                                                 Min Throughput | wait-for-snapshot-create | 4.18309e+07 | byte/s |
|                                                Mean Throughput | wait-for-snapshot-create | 4.18309e+07 | byte/s |
|                                              Median Throughput | wait-for-snapshot-create | 4.18309e+07 | byte/s |
|                                                 Max Throughput | wait-for-snapshot-create | 4.18309e+07 | byte/s |
|                                       100th percentile latency | wait-for-snapshot-create |      619979 |     ms |
|                                  100th percentile service time | wait-for-snapshot-create |      619979 |     ms |
|                                                     error rate | wait-for-snapshot-create |           0 |      % |
|                                                 Min Throughput |                  default |        2.99 |  ops/s |
|                                                Mean Throughput |                  default |        2.99 |  ops/s |
|                                              Median Throughput |                  default |        2.99 |  ops/s |
|                                                 Max Throughput |                  default |           3 |  ops/s |
|                                        50th percentile latency |                  default |       173.7 |     ms |
|                                        90th percentile latency |                  default |     207.338 |     ms |
|                                        99th percentile latency |                  default |     255.386 |     ms |
|                                       100th percentile latency |                  default |     270.333 |     ms |
|                                   50th percentile service time |                  default |     171.866 |     ms |
|                                   90th percentile service time |                  default |      205.81 |     ms |
|                                   99th percentile service time |                  default |     252.939 |     ms |
|                                  100th percentile service time |                  default |      269.73 |     ms |
|                                                     error rate |                  default |           0 |      % |
|                                                 Min Throughput |                    range |           0 |  ops/s |
|                                                Mean Throughput |                    range |           0 |  ops/s |
|                                              Median Throughput |                    range |           0 |  ops/s |
|                                                 Max Throughput |                    range |           0 |  ops/s |
|                                        50th percentile latency |                    range | 4.37725e+07 |     ms |
|                                        90th percentile latency |                    range | 6.10926e+07 |     ms |
|                                        99th percentile latency |                    range | 6.47726e+07 |     ms |
|                                       100th percentile latency |                    range | 6.51695e+07 |     ms |
|                                   50th percentile service time |                    range |      441809 |     ms |
|                                   90th percentile service time |                    range |      466151 |     ms |
|                                   99th percentile service time |                    range |      485194 |     ms |
|                                  100th percentile service time |                    range |      488356 |     ms |
|                                                     error rate |                    range |           0 |      % |
|                                                 Min Throughput |      distance_amount_agg |        2.01 |  ops/s |
|                                                Mean Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                              Median Throughput |      distance_amount_agg |        2.02 |  ops/s |
|                                                 Max Throughput |      distance_amount_agg |        2.03 |  ops/s |
|                                        50th percentile latency |      distance_amount_agg |     4.70118 |     ms |
|                                        90th percentile latency |      distance_amount_agg |     5.24222 |     ms |
|                                        99th percentile latency |      distance_amount_agg |     5.55888 |     ms |
|                                       100th percentile latency |      distance_amount_agg |     5.67187 |     ms |
|                                   50th percentile service time |      distance_amount_agg |     3.11669 |     ms |
|                                   90th percentile service time |      distance_amount_agg |     3.45944 |     ms |
|                                   99th percentile service time |      distance_amount_agg |     3.68248 |     ms |
|                                  100th percentile service time |      distance_amount_agg |     3.77821 |     ms |
|                                                     error rate |      distance_amount_agg |           0 |      % |
|                                                 Min Throughput |            autohisto_agg |           0 |  ops/s |
|                                                Mean Throughput |            autohisto_agg |           0 |  ops/s |
|                                              Median Throughput |            autohisto_agg |           0 |  ops/s |
|                                                 Max Throughput |            autohisto_agg |           0 |  ops/s |
|                                        50th percentile latency |            autohisto_agg | 7.11091e+07 |     ms |
|                                        90th percentile latency |            autohisto_agg | 9.82427e+07 |     ms |
|                                        99th percentile latency |            autohisto_agg |  1.0455e+08 |     ms |
|                                       100th percentile latency |            autohisto_agg | 1.05277e+08 |     ms |
|                                   50th percentile service time |            autohisto_agg |      691756 |     ms |
|                                   90th percentile service time |            autohisto_agg |      726229 |     ms |
|                                   99th percentile service time |            autohisto_agg |      751422 |     ms |
|                                  100th percentile service time |            autohisto_agg |      765487 |     ms |
|                                                     error rate |            autohisto_agg |           0 |      % |
|                                                 Min Throughput |       date_histogram_agg |           0 |  ops/s |
|                                                Mean Throughput |       date_histogram_agg |           0 |  ops/s |
|                                              Median Throughput |       date_histogram_agg |           0 |  ops/s |
|                                                 Max Throughput |       date_histogram_agg |           0 |  ops/s |
|                                        50th percentile latency |       date_histogram_agg |    6.68e+07 |     ms |
|                                        90th percentile latency |       date_histogram_agg | 9.33575e+07 |     ms |
|                                        99th percentile latency |       date_histogram_agg | 9.92378e+07 |     ms |
|                                       100th percentile latency |       date_histogram_agg | 9.98723e+07 |     ms |
|                                   50th percentile service time |       date_histogram_agg |      662967 |     ms |
|                                   90th percentile service time |       date_histogram_agg |      689567 |     ms |
|                                   99th percentile service time |       date_histogram_agg |      700547 |     ms |
|                                  100th percentile service time |       date_histogram_agg |      714373 |     ms |
|                                                     error rate |       date_histogram_agg |           0 |      % |


------------------------------------
[INFO] SUCCESS (took 272942 seconds)
------------------------------------

tlfeng · 2023-03-03T21:52:42Z

Ran the benchmark again after the cache mechanism is added by commit eb78246.

The contender is a modified workload using searchable snapshot feature, with an index backed with remote snapshot. See the PR opensearch-project/opensearch-benchmark-workloads#58 for detail.
The baseline is the original workload, using without using snapshot, with a local index.

Running in an EC2 instance, type c5.18xlarge, Ubuntu 20.04 .
For additional cluster settings, the maximum cache size is set to 30GB by "node.search.cache.size": "30GB", because the full nyc_taxis dataset takes up 20+GB after indexing, and having a larger cache size will have best performance.
The "defaults" provision config instance has got 1GB Java heap size.

opensearch-benchmark compare --baseline=714ceec6-2320-444f-9b9b-e06c078d768a --contender=821c91e8-67fc-465e-915c-5df20fc2e14d

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/


Comparing baseline
  TestExecution ID: 714ceec6-2320-444f-9b9b-e06c078d768a
  TestExecution timestamp: 2023-03-23 06:57:25
  TestProcedure: append-no-conflicts
  ProvisionConfigInstance: defaults

with contender
  TestExecution ID: 821c91e8-67fc-465e-915c-5df20fc2e14d
  TestExecution timestamp: 2023-03-23 05:50:16
  TestProcedure: searchable-snapshot
  ProvisionConfigInstance: defaults

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                        Metric |                     Task |    Baseline |   Contender |     Diff |   Unit |
|--------------------------------------------------------------:|-------------------------:|------------:|------------:|---------:|-------:|
|                    Cumulative indexing time of primary shards |                          |     123.024 |           0 | -123.024 |    min |
|             Min cumulative indexing time across primary shard |                          |     123.024 |           0 | -123.024 |    min |
|          Median cumulative indexing time across primary shard |                          |     123.024 |           0 | -123.024 |    min |
|             Max cumulative indexing time across primary shard |                          |     123.024 |           0 | -123.024 |    min |
|           Cumulative indexing throttle time of primary shards |                          |           0 |           0 |        0 |    min |
|    Min cumulative indexing throttle time across primary shard |                          |           0 |           0 |        0 |    min |
| Median cumulative indexing throttle time across primary shard |                          |           0 |           0 |        0 |    min |
|    Max cumulative indexing throttle time across primary shard |                          |           0 |           0 |        0 |    min |
|                       Cumulative merge time of primary shards |                          |      56.107 |           0 |  -56.107 |    min |
|                      Cumulative merge count of primary shards |                          |         196 |           0 |     -196 |        |
|                Min cumulative merge time across primary shard |                          |      56.107 |           0 |  -56.107 |    min |
|             Median cumulative merge time across primary shard |                          |      56.107 |           0 |  -56.107 |    min |
|                Max cumulative merge time across primary shard |                          |      56.107 |           0 |  -56.107 |    min |
|              Cumulative merge throttle time of primary shards |                          |     9.07342 |           0 | -9.07342 |    min |
|       Min cumulative merge throttle time across primary shard |                          |     9.07342 |           0 | -9.07342 |    min |
|    Median cumulative merge throttle time across primary shard |                          |     9.07342 |           0 | -9.07342 |    min |
|       Max cumulative merge throttle time across primary shard |                          |     9.07342 |           0 | -9.07342 |    min |
|                     Cumulative refresh time of primary shards |                          |     1.08342 |           0 | -1.08342 |    min |
|                    Cumulative refresh count of primary shards |                          |          90 |           0 |      -90 |        |
|              Min cumulative refresh time across primary shard |                          |     1.08342 |           0 | -1.08342 |    min |
|           Median cumulative refresh time across primary shard |                          |     1.08342 |           0 | -1.08342 |    min |
|              Max cumulative refresh time across primary shard |                          |     1.08342 |           0 | -1.08342 |    min |
|                       Cumulative flush time of primary shards |                          |     8.47277 |           0 | -8.47277 |    min |
|                      Cumulative flush count of primary shards |                          |          29 |           1 |      -28 |        |
|                Min cumulative flush time across primary shard |                          |     8.47277 |           0 | -8.47277 |    min |
|             Median cumulative flush time across primary shard |                          |     8.47277 |           0 | -8.47277 |    min |
|                Max cumulative flush time across primary shard |                          |     8.47277 |           0 | -8.47277 |    min |
|                                       Total Young Gen GC time |                          |      47.422 |       48.51 |    1.088 |      s |
|                                      Total Young Gen GC count |                          |       12641 |       12845 |      204 |        |
|                                         Total Old Gen GC time |                          |           0 |           0 |        0 |      s |
|                                        Total Old Gen GC count |                          |           0 |           0 |        0 |        |
|                                                    Store size |                          |     24.1137 |     24.1866 |  0.07285 |     GB |
|                                                 Translog size |                          | 5.12227e-08 |           0 |       -0 |     GB |
|                                        Heap used for segments |                          |           0 |           0 |        0 |     MB |
|                                      Heap used for doc values |                          |           0 |           0 |        0 |     MB |
|                                           Heap used for terms |                          |           0 |           0 |        0 |     MB |
|                                           Heap used for norms |                          |           0 |           0 |        0 |     MB |
|                                          Heap used for points |                          |           0 |           0 |        0 |     MB |
|                                   Heap used for stored fields |                          |           0 |           0 |        0 |     MB |
|                                                 Segment count |                          |          35 |          30 |       -5 |        |
|                                                Min Throughput |                    index |     94798.3 |     95629.7 |  831.409 | docs/s |
|                                               Mean Throughput |                    index |      105198 |      104863 |  -335.14 | docs/s |
|                                             Median Throughput |                    index |      105030 |      103948 | -1082.35 | docs/s |
|                                                Max Throughput |                    index |      119438 |      111944 | -7494.65 | docs/s |
|                                       50th percentile latency |                    index |     511.057 |     512.591 |  1.53382 |     ms |
|                                       90th percentile latency |                    index |      1338.8 |      1338.1 | -0.69279 |     ms |
|                                       99th percentile latency |                    index |     2087.39 |     2077.28 | -10.1052 |     ms |
|                                     99.9th percentile latency |                    index |     2919.39 |     3053.65 |  134.261 |     ms |
|                                    99.99th percentile latency |                    index |     3442.36 |     3556.21 |  113.849 |     ms |
|                                      100th percentile latency |                    index |     3584.06 |     4580.87 |  996.814 |     ms |
|                                  50th percentile service time |                    index |     511.057 |     512.591 |  1.53382 |     ms |
|                                  90th percentile service time |                    index |      1338.8 |      1338.1 | -0.69279 |     ms |
|                                  99th percentile service time |                    index |     2087.39 |     2077.28 | -10.1052 |     ms |
|                                99.9th percentile service time |                    index |     2919.39 |     3053.65 |  134.261 |     ms |
|                               99.99th percentile service time |                    index |     3442.36 |     3556.21 |  113.849 |     ms |
|                                 100th percentile service time |                    index |     3584.06 |     4580.87 |  996.814 |     ms |
|                                                    error rate |                    index |           0 |           0 |        0 |      % |
|                                                Min Throughput | wait-until-merges-finish |     103.696 |   0.0389474 | -103.657 |  ops/s |
|                                               Mean Throughput | wait-until-merges-finish |     103.696 |   0.0389474 | -103.657 |  ops/s |
|                                             Median Throughput | wait-until-merges-finish |     103.696 |   0.0389474 | -103.657 |  ops/s |
|                                                Max Throughput | wait-until-merges-finish |     103.696 |   0.0389474 | -103.657 |  ops/s |
|                                      100th percentile latency | wait-until-merges-finish |     9.35934 |     25675.3 |  25665.9 |     ms |
|                                 100th percentile service time | wait-until-merges-finish |     9.35934 |     25675.3 |  25665.9 |     ms |
|                                                    error rate | wait-until-merges-finish |           0 |           0 |        0 |      % |
|                                                Min Throughput |                  default |     3.01708 |     3.01501 | -0.00208 |  ops/s |
|                                               Mean Throughput |                  default |     3.02779 |     3.02442 | -0.00337 |  ops/s |
|                                             Median Throughput |                  default |     3.02529 |     3.02228 | -0.00301 |  ops/s |
|                                                Max Throughput |                  default |     3.04898 |     3.04304 | -0.00594 |  ops/s |
|                                       50th percentile latency |                  default |     6.84063 |     4.76405 | -2.07658 |     ms |
|                                       90th percentile latency |                  default |     7.82312 |      5.2849 | -2.53822 |     ms |
|                                       99th percentile latency |                  default |     7.99672 |     7.19681 | -0.79992 |     ms |
|                                      100th percentile latency |                  default |     8.01398 |     7.84477 |  -0.1692 |     ms |
|                                  50th percentile service time |                  default |     5.63694 |     3.38259 | -2.25435 |     ms |
|                                  90th percentile service time |                  default |     6.22763 |     3.74197 | -2.48567 |     ms |
|                                  99th percentile service time |                  default |     6.59507 |     5.88154 | -0.71353 |     ms |
|                                 100th percentile service time |                  default |     6.63332 |     6.11312 | -0.52019 |     ms |
|                                                    error rate |                  default |           0 |           0 |        0 |      % |
|                                                Min Throughput |                    range |    0.703365 |    0.618624 | -0.08474 |  ops/s |
|                                               Mean Throughput |                    range |    0.705533 |    0.652529 |   -0.053 |  ops/s |
|                                             Median Throughput |                    range |    0.705033 |    0.656196 | -0.04884 |  ops/s |
|                                                Max Throughput |                    range |    0.709987 |    0.670034 | -0.03995 |  ops/s |
|                                       50th percentile latency |                    range |     197.812 |      439.08 |  241.268 |     ms |
|                                       90th percentile latency |                    range |     203.749 |     444.724 |  240.975 |     ms |
|                                       99th percentile latency |                    range |     205.619 |     451.645 |  246.026 |     ms |
|                                      100th percentile latency |                    range |     206.013 |         454 |  247.987 |     ms |
|                                  50th percentile service time |                    range |     195.642 |     437.047 |  241.406 |     ms |
|                                  90th percentile service time |                    range |     201.148 |     443.052 |  241.903 |     ms |
|                                  99th percentile service time |                    range |     203.296 |     449.791 |  246.496 |     ms |
|                                 100th percentile service time |                    range |     203.715 |     451.673 |  247.958 |     ms |
|                                                    error rate |                    range |           0 |           0 |        0 |      % |
|                                                Min Throughput |      distance_amount_agg |     2.01241 |     2.01194 | -0.00047 |  ops/s |
|                                               Mean Throughput |      distance_amount_agg |     2.02042 |     2.01963 | -0.00079 |  ops/s |
|                                             Median Throughput |      distance_amount_agg |     2.01855 |     2.01784 | -0.00071 |  ops/s |
|                                                Max Throughput |      distance_amount_agg |     2.03673 |     2.03526 | -0.00147 |  ops/s |
|                                       50th percentile latency |      distance_amount_agg |      3.6482 |     3.26822 | -0.37998 |     ms |
|                                       90th percentile latency |      distance_amount_agg |     3.88892 |     3.93898 |  0.05007 |     ms |
|                                       99th percentile latency |      distance_amount_agg |     4.35065 |      4.3716 |  0.02095 |     ms |
|                                      100th percentile latency |      distance_amount_agg |     4.56093 |     4.48179 | -0.07914 |     ms |
|                                  50th percentile service time |      distance_amount_agg |     1.95054 |     1.94054 |    -0.01 |     ms |
|                                  90th percentile service time |      distance_amount_agg |     2.20384 |     2.34373 |  0.13989 |     ms |
|                                  99th percentile service time |      distance_amount_agg |     2.79378 |     2.63273 | -0.16104 |     ms |
|                                 100th percentile service time |      distance_amount_agg |     3.01733 |     2.94395 | -0.07338 |     ms |
|                                                    error rate |      distance_amount_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |            autohisto_agg |     1.50105 |    0.865484 | -0.63557 |  ops/s |
|                                               Mean Throughput |            autohisto_agg |     1.50171 |     1.16175 | -0.33996 |  ops/s |
|                                             Median Throughput |            autohisto_agg |     1.50156 |     1.21269 | -0.28887 |  ops/s |
|                                                Max Throughput |            autohisto_agg |       1.503 |     1.28191 |  -0.2211 |  ops/s |
|                                       50th percentile latency |            autohisto_agg |     461.953 |      529.79 |  67.8366 |     ms |
|                                       90th percentile latency |            autohisto_agg |      473.83 |     7435.69 |  6961.86 |     ms |
|                                       99th percentile latency |            autohisto_agg |         490 |     9005.74 |  8515.74 |     ms |
|                                      100th percentile latency |            autohisto_agg |     507.705 |     9182.42 |  8674.71 |     ms |
|                                  50th percentile service time |            autohisto_agg |     460.948 |     489.107 |  28.1587 |     ms |
|                                  90th percentile service time |            autohisto_agg |     472.264 |     496.943 |   24.679 |     ms |
|                                  99th percentile service time |            autohisto_agg |     489.049 |     503.376 |  14.3265 |     ms |
|                                 100th percentile service time |            autohisto_agg |     506.586 |     505.882 | -0.70386 |     ms |
|                                                    error rate |            autohisto_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |       date_histogram_agg |     1.50256 |     1.50258 |    3e-05 |  ops/s |
|                                               Mean Throughput |       date_histogram_agg |     1.50416 |      1.5042 |    5e-05 |  ops/s |
|                                             Median Throughput |       date_histogram_agg |     1.50379 |     1.50384 |    5e-05 |  ops/s |
|                                                Max Throughput |       date_histogram_agg |     1.50734 |     1.50743 |    9e-05 |  ops/s |
|                                       50th percentile latency |       date_histogram_agg |      491.79 |     468.799 | -22.9906 |     ms |
|                                       90th percentile latency |       date_histogram_agg |     503.322 |     477.032 | -26.2899 |     ms |
|                                       99th percentile latency |       date_histogram_agg |     524.405 |      490.29 | -34.1149 |     ms |
|                                      100th percentile latency |       date_histogram_agg |     526.752 |     499.717 | -27.0353 |     ms |
|                                  50th percentile service time |       date_histogram_agg |     490.616 |     467.579 | -23.0374 |     ms |
|                                  90th percentile service time |       date_histogram_agg |     502.156 |     475.683 | -26.4725 |     ms |
|                                  99th percentile service time |       date_histogram_agg |     523.478 |      489.52 | -33.9585 |     ms |
|                                 100th percentile service time |       date_histogram_agg |     525.856 |     498.355 | -27.5004 |     ms |
|                                                    error rate |       date_histogram_agg |           0 |           0 |        0 |      % |

The below result is running the workload in "test mode"

opensearch-benchmark compare --baseline=1fba59ce-02fc-4fb3-a83f-e76d364587cf --contender=ac7967a0-681f-4520-a0be-1e60accf818b

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/


Comparing baseline
  TestExecution ID: 1fba59ce-02fc-4fb3-a83f-e76d364587cf
  TestExecution timestamp: 2023-03-03 22:10:13
  TestProcedure: searchable_snapshot
  ProvisionConfigInstance: 24gheap

with contender
  TestExecution ID: ac7967a0-681f-4520-a0be-1e60accf818b
  TestExecution timestamp: 2023-03-03 21:22:36
  TestProcedure: searchable_snapshot
  ProvisionConfigInstance: 24gheap

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                        Metric |                Task |    Baseline |   Contender |     Diff |   Unit |
|--------------------------------------------------------------:|--------------------:|------------:|------------:|---------:|-------:|
|                    Cumulative indexing time of primary shards |                     |      0.0131 |           0 |  -0.0131 |    min |
|             Min cumulative indexing time across primary shard |                     |      0.0131 |           0 |  -0.0131 |    min |
|          Median cumulative indexing time across primary shard |                     |      0.0131 |           0 |  -0.0131 |    min |
|             Max cumulative indexing time across primary shard |                     |      0.0131 |           0 |  -0.0131 |    min |
|           Cumulative indexing throttle time of primary shards |                     |           0 |           0 |        0 |    min |
|    Min cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
| Median cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|    Max cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|                       Cumulative merge time of primary shards |                     |           0 |           0 |        0 |    min |
|                      Cumulative merge count of primary shards |                     |           0 |           0 |        0 |        |
|                Min cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|             Median cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|                Max cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|              Cumulative merge throttle time of primary shards |                     |           0 |           0 |        0 |    min |
|       Min cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|    Median cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|       Max cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|                     Cumulative refresh time of primary shards |                     |           0 |           0 |        0 |    min |
|                    Cumulative refresh count of primary shards |                     |           2 |           0 |       -2 |        |
|              Min cumulative refresh time across primary shard |                     |           0 |           0 |        0 |    min |
|           Median cumulative refresh time across primary shard |                     |           0 |           0 |        0 |    min |
|              Max cumulative refresh time across primary shard |                     |           0 |           0 |        0 |    min |
|                       Cumulative flush time of primary shards |                     |           0 |           0 |        0 |    min |
|                      Cumulative flush count of primary shards |                     |           0 |           0 |        0 |        |
|                Min cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|             Median cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|                Max cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|                                       Total Young Gen GC time |                     |           0 |           0 |        0 |      s |
|                                      Total Young Gen GC count |                     |           0 |           0 |        0 |        |
|                                         Total Old Gen GC time |                     |           0 |           0 |        0 |      s |
|                                        Total Old Gen GC count |                     |           0 |           0 |        0 |        |
|                                                    Store size |                     | 1.93715e-07 | 0.000253483 |  0.00025 |     GB |
|                                                 Translog size |                     | 0.000525326 |           0 | -0.00053 |     GB |
|                                        Heap used for segments |                     |           0 |           0 |        0 |     MB |
|                                      Heap used for doc values |                     |           0 |           0 |        0 |     MB |
|                                           Heap used for terms |                     |           0 |           0 |        0 |     MB |
|                                           Heap used for norms |                     |           0 |           0 |        0 |     MB |
|                                          Heap used for points |                     |           0 |           0 |        0 |     MB |
|                                   Heap used for stored fields |                     |           0 |           0 |        0 |     MB |
|                                                 Segment count |                     |           0 |           8 |        8 |        |
|                                                Min Throughput |               index |     5036.44 |     4766.49 | -269.955 | docs/s |
|                                               Mean Throughput |               index |     5036.44 |     4766.49 | -269.955 | docs/s |
|                                             Median Throughput |               index |     5036.44 |     4766.49 | -269.955 | docs/s |
|                                                Max Throughput |               index |     5036.44 |     4766.49 | -269.955 | docs/s |
|                                       50th percentile latency |               index |     182.366 |     192.954 |   10.588 |     ms |
|                                      100th percentile latency |               index |     191.118 |     201.392 |   10.274 |     ms |
|                                  50th percentile service time |               index |     182.366 |     192.954 |   10.588 |     ms |
|                                 100th percentile service time |               index |     191.118 |     201.392 |   10.274 |     ms |
|                                                    error rate |               index |           0 |           0 |        0 |      % |
|                                                Min Throughput |             default |     19.9492 |      17.316 | -2.63324 |  ops/s |
|                                               Mean Throughput |             default |     19.9492 |      17.316 | -2.63324 |  ops/s |
|                                             Median Throughput |             default |     19.9492 |      17.316 | -2.63324 |  ops/s |
|                                                Max Throughput |             default |     19.9492 |      17.316 | -2.63324 |  ops/s |
|                                      100th percentile latency |             default |     55.5903 |     63.5289 |  7.93861 |     ms |
|                                 100th percentile service time |             default |     5.12121 |     5.41401 |   0.2928 |     ms |
|                                                    error rate |             default |           0 |           0 |        0 |      % |
|                                                Min Throughput |               range |     89.3232 |     50.6598 | -38.6634 |  ops/s |
|                                               Mean Throughput |               range |     89.3232 |     50.6598 | -38.6634 |  ops/s |
|                                             Median Throughput |               range |     89.3232 |     50.6598 | -38.6634 |  ops/s |
|                                                Max Throughput |               range |     89.3232 |     50.6598 | -38.6634 |  ops/s |
|                                      100th percentile latency |               range |     14.9984 |     26.1536 |  11.1552 |     ms |
|                                 100th percentile service time |               range |      3.5233 |     6.12207 |  2.59877 |     ms |
|                                                    error rate |               range |           0 |           0 |        0 |      % |
|                                                Min Throughput | distance_amount_agg |     37.3776 |     28.8254 | -8.55211 |  ops/s |
|                                               Mean Throughput | distance_amount_agg |     37.3776 |     28.8254 | -8.55211 |  ops/s |
|                                             Median Throughput | distance_amount_agg |     37.3776 |     28.8254 | -8.55211 |  ops/s |
|                                                Max Throughput | distance_amount_agg |     37.3776 |     28.8254 | -8.55211 |  ops/s |
|                                      100th percentile latency | distance_amount_agg |     31.5522 |     39.5169 |  7.96471 |     ms |
|                                 100th percentile service time | distance_amount_agg |     4.47283 |     4.53224 |  0.05941 |     ms |
|                                                    error rate | distance_amount_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |       autohisto_agg |     60.0009 |     37.5329 |  -22.468 |  ops/s |
|                                               Mean Throughput |       autohisto_agg |     60.0009 |     37.5329 |  -22.468 |  ops/s |
|                                             Median Throughput |       autohisto_agg |     60.0009 |     37.5329 |  -22.468 |  ops/s |
|                                                Max Throughput |       autohisto_agg |     60.0009 |     37.5329 |  -22.468 |  ops/s |
|                                      100th percentile latency |       autohisto_agg |     21.6338 |     33.6817 |   12.048 |     ms |
|                                 100th percentile service time |       autohisto_agg |     4.63413 |     6.72392 |  2.08979 |     ms |
|                                                    error rate |       autohisto_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |  date_histogram_agg |     102.123 |     79.3977 | -22.7255 |  ops/s |
|                                               Mean Throughput |  date_histogram_agg |     102.123 |     79.3977 | -22.7255 |  ops/s |
|                                             Median Throughput |  date_histogram_agg |     102.123 |     79.3977 | -22.7255 |  ops/s |
|                                                Max Throughput |  date_histogram_agg |     102.123 |     79.3977 | -22.7255 |  ops/s |
|                                      100th percentile latency |  date_histogram_agg |     14.5216 |     18.3878 |  3.86622 |     ms |
|                                 100th percentile service time |  date_histogram_agg |     4.43502 |     5.51071 |  1.07569 |     ms |
|                                                    error rate |  date_histogram_agg |           0 |           0 |        0 |      % |

Ran the test with searchable snapshots feature again, which is the new contender, and the baseline is the same as above.
The below is the result. I think cache is not cleaned for the cluster.

opensearch-benchmark compare --baseline=1fba59ce-02fc-4fb3-a83f-e76d364587cf --contender=0e8ab1ca-91c0-49ed-872a-c5cd1427d63e 

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/


Comparing baseline
  TestExecution ID: 1fba59ce-02fc-4fb3-a83f-e76d364587cf
  TestExecution timestamp: 2023-03-03 22:10:13
  TestProcedure: searchable_snapshot
  ProvisionConfigInstance: 24gheap

with contender
  TestExecution ID: 0e8ab1ca-91c0-49ed-872a-c5cd1427d63e
  TestExecution timestamp: 2023-03-03 22:13:26
  TestProcedure: searchable_snapshot
  ProvisionConfigInstance: 24gheap

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                        Metric |                Task |    Baseline |   Contender |     Diff |   Unit |
|--------------------------------------------------------------:|--------------------:|------------:|------------:|---------:|-------:|
|                    Cumulative indexing time of primary shards |                     |      0.0131 |     0.01355 |  0.00045 |    min |
|             Min cumulative indexing time across primary shard |                     |      0.0131 |     0.01355 |  0.00045 |    min |
|          Median cumulative indexing time across primary shard |                     |      0.0131 |     0.01355 |  0.00045 |    min |
|             Max cumulative indexing time across primary shard |                     |      0.0131 |     0.01355 |  0.00045 |    min |
|           Cumulative indexing throttle time of primary shards |                     |           0 |           0 |        0 |    min |
|    Min cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
| Median cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|    Max cumulative indexing throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|                       Cumulative merge time of primary shards |                     |           0 |           0 |        0 |    min |
|                      Cumulative merge count of primary shards |                     |           0 |           0 |        0 |        |
|                Min cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|             Median cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|                Max cumulative merge time across primary shard |                     |           0 |           0 |        0 |    min |
|              Cumulative merge throttle time of primary shards |                     |           0 |           0 |        0 |    min |
|       Min cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|    Median cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|       Max cumulative merge throttle time across primary shard |                     |           0 |           0 |        0 |    min |
|                     Cumulative refresh time of primary shards |                     |           0 | 0.000216667 |  0.00022 |    min |
|                    Cumulative refresh count of primary shards |                     |           2 |           3 |        1 |        |
|              Min cumulative refresh time across primary shard |                     |           0 | 0.000216667 |  0.00022 |    min |
|           Median cumulative refresh time across primary shard |                     |           0 | 0.000216667 |  0.00022 |    min |
|              Max cumulative refresh time across primary shard |                     |           0 | 0.000216667 |  0.00022 |    min |
|                       Cumulative flush time of primary shards |                     |           0 |           0 |        0 |    min |
|                      Cumulative flush count of primary shards |                     |           0 |           0 |        0 |        |
|                Min cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|             Median cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|                Max cumulative flush time across primary shard |                     |           0 |           0 |        0 |    min |
|                                       Total Young Gen GC time |                     |           0 |           0 |        0 |      s |
|                                      Total Young Gen GC count |                     |           0 |           0 |        0 |        |
|                                         Total Old Gen GC time |                     |           0 |           0 |        0 |      s |
|                                        Total Old Gen GC count |                     |           0 |           0 |        0 |        |
|                                                    Store size |                     | 1.93715e-07 | 0.000253686 |  0.00025 |     GB |
|                                                 Translog size |                     | 0.000525326 | 5.12227e-08 | -0.00053 |     GB |
|                                        Heap used for segments |                     |           0 |           0 |        0 |     MB |
|                                      Heap used for doc values |                     |           0 |           0 |        0 |     MB |
|                                           Heap used for terms |                     |           0 |           0 |        0 |     MB |
|                                           Heap used for norms |                     |           0 |           0 |        0 |     MB |
|                                          Heap used for points |                     |           0 |           0 |        0 |     MB |
|                                   Heap used for stored fields |                     |           0 |           0 |        0 |     MB |
|                                                 Segment count |                     |           0 |           8 |        8 |        |
|                                                Min Throughput |               index |     5036.44 |     4780.59 | -255.857 | docs/s |
|                                               Mean Throughput |               index |     5036.44 |     4780.59 | -255.857 | docs/s |
|                                             Median Throughput |               index |     5036.44 |     4780.59 | -255.857 | docs/s |
|                                                Max Throughput |               index |     5036.44 |     4780.59 | -255.857 | docs/s |
|                                       50th percentile latency |               index |     182.366 |     192.425 |   10.059 |     ms |
|                                      100th percentile latency |               index |     191.118 |      200.23 |  9.11209 |     ms |
|                                  50th percentile service time |               index |     182.366 |     192.425 |   10.059 |     ms |
|                                 100th percentile service time |               index |     191.118 |      200.23 |  9.11209 |     ms |
|                                                    error rate |               index |           0 |           0 |        0 |      % |
|                                                Min Throughput |             default |     19.9492 |     21.2576 |  1.30837 |  ops/s |
|                                               Mean Throughput |             default |     19.9492 |     21.2576 |  1.30837 |  ops/s |
|                                             Median Throughput |             default |     19.9492 |     21.2576 |  1.30837 |  ops/s |
|                                                Max Throughput |             default |     19.9492 |     21.2576 |  1.30837 |  ops/s |
|                                      100th percentile latency |             default |     55.5903 |     52.5947 | -2.99563 |     ms |
|                                 100th percentile service time |             default |     5.12121 |     5.22946 |  0.10825 |     ms |
|                                                    error rate |             default |           0 |           0 |        0 |      % |
|                                                Min Throughput |               range |     89.3232 |     75.9239 | -13.3993 |  ops/s |
|                                               Mean Throughput |               range |     89.3232 |     75.9239 | -13.3993 |  ops/s |
|                                             Median Throughput |               range |     89.3232 |     75.9239 | -13.3993 |  ops/s |
|                                                Max Throughput |               range |     89.3232 |     75.9239 | -13.3993 |  ops/s |
|                                      100th percentile latency |               range |     14.9984 |     17.3466 |  2.34822 |     ms |
|                                 100th percentile service time |               range |      3.5233 |     3.87252 |  0.34922 |     ms |
|                                                    error rate |               range |           0 |           0 |        0 |      % |
|                                                Min Throughput | distance_amount_agg |     37.3776 |     32.2081 | -5.16943 |  ops/s |
|                                               Mean Throughput | distance_amount_agg |     37.3776 |     32.2081 | -5.16943 |  ops/s |
|                                             Median Throughput | distance_amount_agg |     37.3776 |     32.2081 | -5.16943 |  ops/s |
|                                                Max Throughput | distance_amount_agg |     37.3776 |     32.2081 | -5.16943 |  ops/s |
|                                      100th percentile latency | distance_amount_agg |     31.5522 |     36.5585 |  5.00635 |     ms |
|                                 100th percentile service time | distance_amount_agg |     4.47283 |     5.15472 |  0.68189 |     ms |
|                                                    error rate | distance_amount_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |       autohisto_agg |     60.0009 |     54.1452 | -5.85571 |  ops/s |
|                                               Mean Throughput |       autohisto_agg |     60.0009 |     54.1452 | -5.85571 |  ops/s |
|                                             Median Throughput |       autohisto_agg |     60.0009 |     54.1452 | -5.85571 |  ops/s |
|                                                Max Throughput |       autohisto_agg |     60.0009 |     54.1452 | -5.85571 |  ops/s |
|                                      100th percentile latency |       autohisto_agg |     21.6338 |     23.4731 |  1.83932 |     ms |
|                                 100th percentile service time |       autohisto_agg |     4.63413 |     4.67072 |  0.03659 |     ms |
|                                                    error rate |       autohisto_agg |           0 |           0 |        0 |      % |
|                                                Min Throughput |  date_histogram_agg |     102.123 |      80.833 | -21.2902 |  ops/s |
|                                               Mean Throughput |  date_histogram_agg |     102.123 |      80.833 | -21.2902 |  ops/s |
|                                             Median Throughput |  date_histogram_agg |     102.123 |      80.833 | -21.2902 |  ops/s |
|                                                Max Throughput |  date_histogram_agg |     102.123 |      80.833 | -21.2902 |  ops/s |
|                                      100th percentile latency |  date_histogram_agg |     14.5216 |     18.1249 |  3.60327 |     ms |
|                                 100th percentile service time |  date_histogram_agg |     4.43502 |     5.41206 |  0.97704 |     ms |
|                                                    error rate |  date_histogram_agg |           0 |           0 |        0 |      % |

andrross added enhancement Enhancement or improvement to existing feature or request untriaged Indexing & Search and removed untriaged labels Oct 14, 2022

andrross mentioned this issue Oct 14, 2022

[META] Searchable Snapshot Tracking Issue #2919

Closed

22 tasks

anasalkouz added this to Searchable Snapshots Oct 14, 2022

anasalkouz moved this to Todo in Searchable Snapshots Oct 14, 2022

anasalkouz assigned tlfeng Oct 17, 2022

tlfeng closed this as completed Nov 4, 2022

Repository owner moved this from Todo to Done in Searchable Snapshots Nov 4, 2022

tlfeng mentioned this issue Nov 9, 2022

[BUG][Searchable Snapshot] Failed to run 'range' query in a large index with error "SearchPhaseExecutionException: all shards failed" through OpenSearch-Benchmark #5172

Closed

andrross mentioned this issue Nov 11, 2022

[Searchable Snapshot] Capture initial CPU profiling information #5217

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Searchable Snapshot] Configure and run initial benchmarks #4797

[Searchable Snapshot] Configure and run initial benchmarks #4797

andrross commented Oct 14, 2022

tlfeng commented Oct 25, 2022 •

edited

Loading

andrross commented Oct 27, 2022

tlfeng commented Oct 31, 2022 •

edited

Loading

tlfeng commented Nov 3, 2022 •

edited

Loading

tlfeng commented Nov 4, 2022 •

edited

Loading

tlfeng commented Nov 4, 2022 •

edited

Loading

tlfeng commented Nov 4, 2022 •

edited

Loading

tlfeng commented Nov 10, 2022 •

edited

Loading

tlfeng commented Mar 3, 2023 •

edited

Loading

[Searchable Snapshot] Configure and run initial benchmarks #4797

[Searchable Snapshot] Configure and run initial benchmarks #4797

Comments

andrross commented Oct 14, 2022

tlfeng commented Oct 25, 2022 • edited Loading

andrross commented Oct 27, 2022

tlfeng commented Oct 31, 2022 • edited Loading

tlfeng commented Nov 3, 2022 • edited Loading

tlfeng commented Nov 4, 2022 • edited Loading

tlfeng commented Nov 4, 2022 • edited Loading

tlfeng commented Nov 4, 2022 • edited Loading

tlfeng commented Nov 10, 2022 • edited Loading

tlfeng commented Mar 3, 2023 • edited Loading

tlfeng commented Oct 25, 2022 •

edited

Loading

tlfeng commented Oct 31, 2022 •

edited

Loading

tlfeng commented Nov 3, 2022 •

edited

Loading

tlfeng commented Nov 4, 2022 •

edited

Loading

tlfeng commented Nov 4, 2022 •

edited

Loading

tlfeng commented Nov 4, 2022 •

edited

Loading

tlfeng commented Nov 10, 2022 •

edited

Loading

tlfeng commented Mar 3, 2023 •

edited

Loading