Update docs to hide Mesos

[w-gao]

Closes #4354.

Changelog Entry

To be copied to the draft changelog by merger:

• Update docs to hide Mesos

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passes tests.
• Make sure the PR has been reviewed since its last modification. If not, review it.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.

[w-gao]

It looks like our readthedocs build is failing after #4361: https://readthedocs.org/projects/toil/builds/19574176/.

I think we need to update the build steps to include the new autoapi dependency. Also, it looks like our actual build is still using sphinx<2 while we have sphinx>=4,<6 in requirements-dev.txt.

[adamnovak]

What's here looks fine, but there are several more places in the docs we want to touch if we want a more cohesive use of Kubernetes. For example:

• There's a bunch of examples using Mesos in

  toil/docs/appendices/deploy.rst

  Line 73 in 1f0930b

  $ python main.py --batchSystem=mesos …

• The autoscaling documentation and examples in

  toil/docs/running/cloud/amazon.rst

  Line 109 in 36b54c4

  **TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:latest** --- This is optional. It specifies a mesos docker image that we maintain with the latest version of toil installed on it. If you want to use a different version of toil, please specify the image tag you need from https://quay.io/repository/ucsc_cgl/toil?tag=latest&tab=tags.

  talk about Mesos and we even have some lingering "Mesos slaves".
• The getting started examples in

  toil/docs/gettingStarted/quickStart.rst

  Line 35 in 1f0930b

  Toil supports many different batch systems such as `Apache Mesos`_ and Grid Engine; its versatility makes it

  steer users towards Mesos

I think we want to get those, and in general all the places in the docs where we use Mesos when there's not a good reason to pick it over Kubernetes.

[adamnovak]

Should this one go first? Otherwise without the option users get both warnings.

[w-gao]

I originally intended to show both warnings if the option isn't set, but it might be redundant now that I look at it.

[w-gao]

I forgot to replace this in the last PR.

[w-gao]

Does the kubernetes batch system not support autoscaling? I tried this with --clusterType=kubernetes, but when I run:

python sort.py aws:us-west-2:<job-store-name> \
          --provisioner aws \
          --batchSystem kubernetes \
          --nodeTypes t2.medium \
          --maxNodes 2

I get:

[2023-03-24T23:04:25+0000] [scaler    ] [E] [toil.provisioners.clusterScaler] Exception encountered in scaler thread. Making a best-effort attempt to keep going, but things may go wrong from now on.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/toil/provisioners/clusterScaler.py", line 1149, in tryRun
    self.scaler.updateClusterSize(estimatedNodeCounts)
  File "/usr/local/lib/python3.10/dist-packages/toil/provisioners/clusterScaler.py", line 754, in updateClusterSize
    newNodeCount = self.setNodeCount(instance_type, estimatedNodeCount, preemptible=nodeShape.preemptible)
  File "/usr/local/lib/python3.10/dist-packages/toil/provisioners/clusterScaler.py", line 802, in setNodeCount
    raise RuntimeError('Non-scalable batch system abusing a scalable-only function.')
RuntimeError: Non-scalable batch system abusing a scalable-only function.

It does look like the kubernetes batch system doesn't implement AbstractScalableBatchSystem.

Are there other ways to dynamically spin up nodes that I am not aware of?

[adamnovak]

It looks like our Kubernetes implementation supports scaling up and down through Kubernetes's cluster autoscaler (which you get when you pass a range of numbers instead of a single number for the number of nodes you want of a given type, when launching the cluster).

Running the Toil-integrated autoscaler as part of the workflow needs the AbstractScalableBatchSystem functions to e.g. drain nodes to safely scale them away. The Kubernetes cluster autoscaler is supposed to take care of all of that inside of Kubernetes and not inside the individual workflows.

Now I am working on this

[adamnovak]

@DailyDreaming says I can just merge this without another review.