Skip to content

Commit

Permalink
Switch to native OpenAPI (opensearch-project#208)
Browse files Browse the repository at this point in the history
* ## Switching over to native OpenAPI:
- Removed Smithy:
  - Removed Smithy models (`model` folder)
  - Removed Gradle
  - Removed legacy test framework
  - Removed build-openapi-specs workflow
- Added Editable OpenAPI spec:
  - Added multi-file OpenApi spec (`spec` folder)
  - Added a tool to merge multi-file spec into single-file spec
  - Added build-single-file-specs workflow
- Updated Documents
## Adding Request and Response Bodies

The OpenAPI spec has also been updated with the request and response bodies that we extracted from ElasticSearch OpenAPI spec. This is done within the same PR as the native OpenAPI switch because we also used the ElasticSearch OpenAPI spec to map the query/path schemas to the schemas of many components in the bodies.

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Removed coverage.yml

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Removed references to Xpack

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Removed hard-coded params for merger

Signed-off-by: Theo Truong <theotr@amazon.com>

---------

Signed-off-by: Theo Truong <theotr@amazon.com>
  • Loading branch information
nhtruong authored Apr 1, 2024
1 parent c256d99 commit cd96458
Show file tree
Hide file tree
Showing 644 changed files with 85,398 additions and 60,876 deletions.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Build OpenAPI specs
name: Build latest single-file specs
on: [push, pull_request]

jobs:
Expand All @@ -8,17 +8,24 @@ jobs:
- name: Checkout the repo
uses: actions/checkout@v3

- name: Setup Node.js
uses: actions/setup-node@v2
with:
node-version: '20'

- name: Build
run: ./gradlew build

- name: Move OpenAPI specs to root
run: mv build/smithyprojections/opensearch-api-specification/full/openapi/OpenSearch.openapi.json .
working-directory: ./tools
run: |-
npm install
export ROOT_PATH=../spec/OpenSearch.openapi.yaml
export OUTPUT_PATH=../builds/OpenSearch.latest.yaml
npm run merge -- $ROOT_PATH $OUTPUT_PATH
- name: Upload OpenAPI model artifact
uses: actions/upload-artifact@v3
with:
name: opensearch-openapi
path: ./OpenSearch.openapi.json
path: ./builds/OpenSearch.latest.yaml

- name: GitHub App token
id: github_app_token
Expand All @@ -34,9 +41,9 @@ jobs:
if: github.repository == 'opensearch-project/opensearch-api-specification' && github.event_name == 'push' && github.event.ref == 'refs/heads/main'
with:
token: ${{ steps.github_app_token.outputs.token }}
commit-message: Update OpenAPI specs
commit-message: Build latest single-file specs
signoff: true
title: Update OpenAPI specs
title: Build latest single-file specs
body: |
I've noticed an update to the models. This pull request updates the OpenAPI specs accordingly.
branch: create-pull-request/specs-update
I've noticed an update in the `./spec` folder. This pull request updates `./build/OpenSearch.latest.yaml` accordingly.
branch: create-pull-request/single-file-specs
46 changes: 0 additions & 46 deletions .github/workflows/ci.yaml

This file was deleted.

55 changes: 0 additions & 55 deletions .github/workflows/coverage.yml

This file was deleted.

18 changes: 0 additions & 18 deletions .github/workflows/domain-check.sh

This file was deleted.

51 changes: 7 additions & 44 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,48 +1,11 @@
# smithy files
bin/
# Mac OS
.DS_Store

# Node.js
node_modules

# intellij files
.idea/
*.iml
*.ipr
*.iws
build-idea/
out/

# include shared intellij config
!.idea/inspectionProfiles/Project_Default.xml
!.idea/runConfigurations/Debug_OpenSearch.xml
!.idea/vcs.xml

# These files are generated in the main tree by IntelliJ
benchmarks/src/main/generated/*

# eclipse files
.project
.classpath
.settings
build-eclipse/

# netbeans files
nb-configuration.xml
nbactions.xml

# gradle stuff
.gradle/
build/

# vscode stuff
.vscode/

# testing stuff
**/.local*
.vagrant/
/logs/

# osx stuff
.DS_Store

.smithy.lsp.log

# Javascript stuff
node_modules
# VSCode files
.vscode/
15 changes: 3 additions & 12 deletions CLIENT_GENERATOR_GUIDE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Generate Clients for OpenSearch using OpenAPI Specification

OpenSearch Clients are available in multiple programming languages. The biggest challenge with this is keeping the clients up to date with the latest changes in OpenSearch. To solve this problem, we're automating the process of generating clients for OpenSearch using the OpenAPI specification. While OpenAPI comes with many well established off-the-shelf client generators for most languages, the OpenSearch APIs come with a lot of quirkiness that makes it near impossible to use these off-the-shelf generators. For this reason, we've opted to write our own client generators that is specifically tailored to the OpenSearch APIs. This document will walk you through the process of generating clients from [OpenSearch OpenAPI spec](./OpenSearch.openapi.json), more specifically client API methods.
OpenSearch Clients are available in multiple programming languages. The biggest challenge with this is keeping the clients up to date with the latest changes in OpenSearch. To solve this problem, we're automating the process of generating clients for OpenSearch using the OpenAPI specification. While OpenAPI comes with many well established off-the-shelf client generators for most languages, the OpenSearch APIs come with a lot of quirkiness that makes it near impossible to use these off-the-shelf generators. For this reason, we've opted to write our own client generators that is specifically tailored to the OpenSearch APIs. This document will walk you through the process of generating clients from [OpenSearch OpenAPI spec](builds/OpenSearch.latest.yaml), more specifically client API methods.

## The Grouping of API Operations
The OpenSearch clients, though written in different languages for different frameworks, share one distinct characteristic: API Operations are grouped into actions and namespaces. Operations serving the same functionality are often grouped together into a single API Action, which is represented in the client as an API method.
Expand All @@ -17,7 +17,7 @@ The **indices.get_field_mapping**, is consisted of 2 operations:

In a client, the `search` operations are grouped in to a single API method, `client.search(...)`, and the `indices.get_field_mapping` operations are grouped into a single API method, `get_field_mapping` of the `indices` namespace, `client.indices.get_field_mapping(...)`

In the [OpenAPI spec](./OpenSearch.openapi.json), this grouping is denoted by `x-operation-group` vendor extension in every operation definition. The value of this extension is the name of the API action (like `search` or `indices.get_field_mapping`). Operations with the same `x-operation-group` value are guaranteed to have the same query string parameters, response body, and request body (for PUT/POST/DELETE operations). Common path parameters are also guaranteed to be the same. The only differences between operations are the HTTP method and the path. With that in mind, below are rules on how to combine operations of different HTTP methods and path compositions.
In the [OpenAPI spec](./builds/OpenSearch.latest.yaml), this grouping is denoted by `x-operation-group` vendor extension in every operation definition. The value of this extension is the name of the API action (like `search` or `indices.get_field_mapping`). Operations with the same `x-operation-group` value are guaranteed to have the same query string parameters, response body, and request body (for PUT/POST/DELETE operations). Common path parameters are also guaranteed to be the same. The only differences between operations are the HTTP method and the path. With that in mind, below are rules on how to combine operations of different HTTP methods and path compositions.

- If an operation is marked with `x-ignorable: "true"`, then ignore the operation. Such an operation has been deprecated and has been replaced by a newer one. As far as the clients are concerned, ignorable operations do not exist.
- If two operations have identical HTTP methods, but different paths: use the path that best matches the path parameters provided.
Expand All @@ -41,17 +41,8 @@ def search(self, index=None, body=None):
return self.perform_request(method, path, body=body)
```

## Handling Path Parameters

### List Type
Path parameters in OpenSearch sometimes allow comma-seperated values. For example, the `index` parameter in `GET /{index}/_search` can either have the value of a single index, or a comma-seperated list of indices like `books,movies`. However, Smithy, the tool we use to draft OpenAPI spec, does not support list as a data-type for path parameters. To work around this, we use the `x-data-type: "list"` vendor extension to denote that a path parameter can be a comma-seperated list of values. The client, when receiving an array of values for a path parameter of type list, will join the values with a comma.

### Enum Options
Path parameters are sometimes restricted to certain enum values. However, since Smithy does not allow enum type or enum-list type for path parameters, we use the `x-enum-options` vendor extension to denote the enum values that are allowed for a path parameter. As of right now, the clients do not validate the values of path parameters against the enum values, however, the list of these values are often written into the parameter description.

Path parameters can also come with `x-string-pattern` vendor extension to denote a regex pattern that the parameter value must match. The clients do NOT validate the values of path parameters against the regex pattern, however. So, it's safe to ignore this extension.

### Overloaded Name
## Overloaded Name
You will also encounter `x-overloaded-param: "metric"` for the `node_id` path parameter of `GET /_nodes/{node_id}` operation in `nodes.info` action. This is a special case where the path parameter is overloaded to accept either a node ID or a metric name. The `client.nodes.info` method when called with either `metric` or `node_id` (but not both), will use `GET /_nodes/{node_id}` operation (even though the path parameter name is `node_id`). When called with both `metric` and `node_id`, it will use `GET /_nodes/{node_id}/{metric}` operation.

## Handling Bulk Operations
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ As with other types of contributions, the first step is to [open an issue on Git

OpenSearch is an open source product released under the Apache 2.0 license (see either [the Apache site](https://www.apache.org/licenses/LICENSE-2.0) or the [LICENSE.txt file](LICENSE.txt)). The Apache 2.0 license allows you to freely use, modify, distribute, and sell your own products that include Apache 2.0 licensed software.

We respect intellectual property rights of others and we want to make sure all incoming contributions are correctly attributed and licensed. A Developer Certificate of Origin (DCO) is a lightweight mechanism to do that.
We respect intellectual property rights of others, and we want to make sure all incoming contributions are correctly attributed and licensed. A Developer Certificate of Origin (DCO) is a lightweight mechanism to do that.

The DCO is a declaration attached to every contribution made by every developer. In the commit message of the contribution, the developer simply adds a `Signed-off-by` statement and thereby agrees to the DCO, which you can find below or at [DeveloperCertificate.org](http://developercertificate.org/).

Expand Down
Loading

0 comments on commit cd96458

Please sign in to comment.