Task Management Experimental Status

[henningandersen]

This is an experimental status issue, documenting the known remains before the Task management API can move out of experimental/beta state.

The primary outstanding items on the API is cementing the intentions of the API usage and its backwards compatibility guarantees. This was discussed in some detail here. This in turn adds work in several areas.

Following are the expected remaining actions before removing the experimental/beta status:

• Reframe the tasks API as a powerful troubleshooting tool that should not be used for building applications (discuss + docs).
• Clarify in docs that only the basic structure of the tasks API follows our backwards compatibliity principles. The structure (parent/child) of tasks, action names, descriptions, status and more can change in any release, including potentially micro/bugfix.
• Remove node and id from response (Remove node and id from list task response #31253)
• Ensure we do not use ListTasksResponse as response type for rethrottle (part of Reindex resiliency #42612)
• Add explicit cancel APIs where appropriate to not point users to the task cancel API.
• Similarly add explicit APIs for obtaining the result of tasks submitted with wait_for_completion=false (at least update and delete-by-query) and a mechanism for cleaning up older results once no longer needed.
• Remove the ability to view persisted task results, since these should be obtained through dedicated APIs (notice: the ephemeral task id is inherently unsafe to use, since it may be reused after node restarts).
• (optional) Add manage_tasks security privilege.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Task Management)

[amitelastic]

is it still in beta/experimental state?

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[devoxel]

The documentation for async delete by has the note:

When you are done with a task, you should delete the task document so Elasticsearch can reclaim the space.

But direct access to hidden indices like .tasks is now deprecated.

I see there's a TODO item in the issue:

Similarly add explicit APIs for obtaining the result of tasks submitted with wait_for_completion=false (at least update and delete-by-query) and a mechanism for cleaning up older results once no longer needed.

How is it intended to key these lookups? If the task id is inherently unsafe to use because it doesn't survive node restart, will the lookup on the result of a submitted task be resilient here?

[szabado-faire]

Reframe the tasks API as a powerful troubleshooting tool that should not be used for building applications (discuss + docs).

The task API is currently the best way I've come across to trigger expensive or slow operations (eg. reindex, force merge) and then wait for them to complete in a durable way. A blocking API call is more brittle, and does not survive server restarts or other similar disruptions.

Is there a proposed alternative if tasks are instead supposed to be a troubleshooting tool?