Added ML Model APIs

[nathaliellenaa]

Description

Added missing ML Model APIs.

Issues Resolved

Part of opensearch-project/opensearch-py#867.

ML Model APIs to add

• GET /_plugins/_ml/models/{model_id}
• POST /_plugins/_ml/models/meta
• POST /_plugins/_ml/models/_register_meta
• POST /_plugins/_ml/models/_search
• POST /_plugins/_ml/models/_undeploy
• POST /_plugins/_ml/models/_unload
• POST /_plugins/_ml/models/{model_id}/_unload
• POST /_plugins/_ml/models/_upload
• POST /_plugins/_ml/models/{model_id}/_load
• POST /_plugins/_ml/models/{model_id}/_predict
• POST /_plugins/_ml/models/{model_id}/chunk/{chunk_number}
• POST /_plugins/_ml/models/{model_id}/upload_chunk/{chunk_number}
• PUT /_plugins/_ml/models/{model_id}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Changes Analysis

Commit SHA: 1e2619d
Comparing To SHA: a3849b4

API Changes

Summary

├─┬Paths
│ ├──[➕] path (4992:3)
│ ├──[➕] path (4860:3)
│ ├──[➕] path (5066:3)
│ ├──[➕] path (4895:3)
│ ├──[➕] path (5084:3)
│ ├──[➕] path (4974:3)
│ ├──[➕] path (4813:3)
│ ├──[➕] path (5046:3)
│ ├──[➕] path (4876:3)
│ ├──[➕] path (5026:3)
│ ├─┬/_plugins/_ml/models/{model_id}/_undeploy
│ │ └─┬POST
│ │   └──[➕] requestBody (27722:7)❌ 
│ ├─┬/_plugins/_ml/models/{model_id}
│ │ ├──[➕] get (4930:7)
│ │ └──[➕] put (4944:7)
│ ├─┬/_plugins/_ml/models/_search
│ │ └──[➕] post (4846:7)
│ └─┬/_plugins/_ml/models/_register
│   └─┬POST
│     └─┬Requestbody
│       └─┬application/json
│         └─┬Schema
│           ├──[➖] required (27264:17)❌ 
│           ├─┬model_format
│           │ └──[🔀] $ref (33875:13)❌ 
│           ├─┬model_group_id
│           │ └──[🔀] $ref (33654:13)❌ 
│           └─┬version
│             └──[🔀] $ref (35229:13)❌ 
└─┬Components
  ├──[➕] requestBodies (27482:7)
  ├──[➕] requestBodies (27722:7)
  ├──[➕] requestBodies (27738:7)
  ├──[➕] requestBodies (27545:7)
  ├──[➕] requestBodies (27784:7)
  ├──[➕] requestBodies (27754:7)
  ├──[➕] requestBodies (27795:7)
  ├──[➕] requestBodies (27439:7)
  ├──[➕] requestBodies (27638:7)
  ├──[➕] responses (31031:7)
  ├──[➕] responses (30890:7)
  ├──[➕] responses (31013:7)
  ├──[➕] responses (31007:7)
  ├──[➕] responses (30928:7)
  ├──[➕] responses (30803:7)
  ├──[➕] responses (30824:7)
  ├──[➕] responses (30902:7)
  ├──[➕] responses (31019:7)
  ├──[➕] responses (30955:7)
  ├──[➕] parameters (23237:7)
  ├──[➕] parameters (23183:7)
  ├──[➕] parameters (23201:7)
  ├──[➕] parameters (23135:7)
  ├──[➕] parameters (23249:7)
  ├──[➕] parameters (23256:7)
  ├──[➕] parameters (23195:7)
  ├──[➕] parameters (23243:7)
  ├──[➕] parameters (23128:7)
  ├──[➕] schemas (54911:7)
  ├──[➕] schemas (55236:7)
  ├──[➕] schemas (55244:7)
  ├──[➕] schemas (54629:7)
  ├──[➕] schemas (54598:7)
  ├──[➕] schemas (54706:7)
  ├──[➕] schemas (54945:7)
  ├──[➕] schemas (54868:7)
  ├──[➕] schemas (55249:7)
  ├──[➕] schemas (54904:7)
  ├──[➕] schemas (55241:7)
  ├──[➕] schemas (33875:7)
  └─┬ml._common___Source
    ├─┬model_format
    │ └──[🔀] $ref (33875:13)❌ 
    └─┬model_state
      └──[➕] enum (55068:15)

Document Element	Total Changes	Breaking Changes
paths	18	5
components	42	1

• ❌ BREAKING Changes: 6 out of 60
• Modifications: 4
• Removals: 1
• Additions: 55
• Breaking Removals: 1
• Breaking Modifications: 4
• Breaking Additions: 1

Report

The full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12564099744/artifacts/2375148771

API Coverage

	Before	After	Δ
Covered (%)	609 (59.65 %)	622 (60.92 %)	13 (1.27 %)
Uncovered (%)	412 (40.35 %)	399 (39.08 %)	-13 (-1.27 %)
Unknown	43	43	0

[github-actions]

Spec Test Coverage Analysis

Total	Tested
546	547 (100.18 %)

[dblock]

Looking good. I have some smallish nits. Iterate to green (validation is failing, etc.)

[dblock]

Btw, there's a whole bunch of others missing per #168. They don't all need to be done at the same time.

[nathaliellenaa]

Yes, I'm planning to create some small PRs based on the APIs category (model, model groups, connector, agents, etc.) I will also add more missing Model APIs to this PR.

[dhrubo-os]

DCO is missing @nathaliellenaa

[dblock]

Looking good. Some nits below.

In tests try to get rid of multiple-paths-detected: false as much as possible by moving setup into prologues and teardown into epilogues. You should only need that set if there's a need to call an unrelated API (e.g. wait on a task) to complete the test. Otherwise chapters should only have chapters for the API being tested.

[nathaliellenaa]

The APIs POST /_plugins/_ml/models/{model_id}/{version}/_register and POST /_plugins/_ml/models/{model_id}/{version}/_upload will be deprecated since cx are now only using the POST /_plugins/_ml/models/_register API. Both APIs were non-functional in current and previous versions as well. I believe we should refrain from adding these APIs to the specification to avoid future deprecation efforts in the client side.

[dblock]

Good work.

Go through tests and make sure only the API being tested is in the chapters and everything else is in prologues or epilogues, and that the naming is consistent.

[dblock]

Looks like tests fail consistently in CI, https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12520890851/job/34927206366?pr=733, maybe needs longer retries or maybe there's a real problem.

[dblock]

To be consistent tests/plugins/ml/models should be tests/plugins/ml/ml/models. I know the double ml looks weird there, but the folder ml is the name of the test suite (like default) and from there the path is the API path, and all the APIs being tested are under _ml/models.

[nathaliellenaa]

Looks like tests fail consistently in CI, https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12520890851/job/34927206366?pr=733, maybe needs longer retries or maybe there's a real problem.

The tests passed when get_completed_deploy_model_task is moved to chapters section, but they are failing when it is moved to prologues section. I'll look deeper into this issue.

Update: It seems that when the deploy task state is still CREATED, it immediately predicts the model without retrying it until the state is RUNNING or COMPLETED. I suspected that it's possibly picking up the state from registering a model previously, but I already verified that get_completed_deploy_model_task used the correct deploy model task id and task type.

@dblock Do you have any idea what causes this issue?

[INFO] => GET /_plugins/_ml/tasks/6I1wGZQBkjzF5RCKw0hU ({}) [application/json] 
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "model_id": "541wGZQBkjzF5RCKdUiw",
  "task_type": "DEPLOY_MODEL",
  "function_name": "TEXT_EMBEDDING",
  "state": "CREATED",
  "worker_node": [
    "kLV6JVKsRpCfUABzKFFbsQ",
    "Wwv5YlwPRCeqFoTtk0-3Cg",
    "1Mel2oTKSH2Sxvy5W16QvQ",
    "VodBQqJ-TluOEBf4o5h72Q"
  ],
  "create_time": 1735593608020,
  "last_update_time": 1735593608020,
  "is_async": true
}
[INFO] $ {
  "outputs": {
    "model_id": "541wGZQBkjzF5RCKdUiw"
  }
}
[INFO] => POST /_plugins/_ml/models/541wGZQBkjzF5RCKdUiw/_predict ({}) [application/json] {
  "query_text": "The best selling book series in history is Harry Potter",
  "text_docs": [
    "Harry Potter is written by J.K. Rowling",
    "The Great Gatsby is a story of wealth and tragedy",
    "The Lord of the Rings is an epic high fantasy novel",
    "The best selling book series in history is Harry Potter"
  ]
}
[INFO] <= 400 (application/json) | {
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Model not ready yet. Please deploy the model first."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Model not ready yet. Please deploy the model first."
  },
  "status": 400
}

[dblock]

I don't believe retry can do this:

      until:
        - path: payload.state
          equal: COMPLETED

Retry schema is here. You want to wait till the response payload is COMPLETED, so:

response:
  payload:
    state: COMPLETED

Then retry will keep retrying until that's true.

    retry:
      count: 3
      wait: 10000

There's a bug in the schema, it should have prevented you from adding until which doesn't exist loudly. Open a bug please, and feel free to fix.

[nathaliellenaa]

Retry schema is here. You want to wait till the response payload is COMPLETED, so:

response:
  payload:
    state: COMPLETED

I checked the error and it still has the same issue on the get_completed_deploy_model_task, where it immediately predicts the model without waiting until deploy task state is COMPLETED. It seems that the prologues doesn't have response property (ref). Should the get_completed_deploy_model_task be moved to the chapters section then?

[dblock]

It seems that the prologues doesn't have response property (ref). Should the get_completed_deploy_model_task be moved to the chapters section then?

I think the right fix would be to add support for payload properties into the prologue. Want to give it a shot?