Draft Changelog

The following items should be added to the changelog for the next Toil release, at which point the list should be cleared. Add new items at the bottom:

Version 7.x Draft

Changelog

Highlighted Features Added

• toil debug-job now has --retrieveTaskDirectory <dir> which will set up a job's downloaded files under <dir> and try to stop the job after doing the downloads. Jobs can call self.files_downloaded_hook() to provide a stopping point for this mode. (#4815)
• toil debug-job can now reconstruct the inside-the-container environment for CWL and WDL tasks. (#4815)
• Added support for caching on Slurm and other HPC schedulers (#4775)
• Replace all instances of boto2 with boto3 for all Toil AWS code (#4718)
• Add support for Python 3.12 (#4718)
• Add support for Python 3.13 (#5145)
• Ceph input/output errors from file locking functions are now tolerated. (#4874)
• Toil now uses flock to enable directory locks to work properly (#4924)
• Added support to get Slurm partitions and automatically send jobs to GPUs on Slurm (#4833) (supports both CWL and WDL)
• New --symlinkJobStoreReads=False option lets you force local-node copies (possibly in the cache) even when reading directly from a FileJobStore is possible, potentially reducing shared filesystem IO. (#4673)
• Toil now supports reading and writing MiniWDL's call cache. (#4797)
• Toil now supports running CWL and WDL workflows from Dockstore, by using either a Dockstore page URL or TRS ID as the URL/filename of the workflow to run. Since these often contain ? or #, remember to quote them on the command line! (#5049)
• Add support for parallel file imports (#5114)
  • New argument --importWorkersThreshold. This specifies the threshold where files will begin to be imported on individual jobs. Small files will be batched into the same import job up to this threshold.
  • --importWorkersDisk defaults to 1 MiB. Should be increased when download streaming is not possible on a worker.

Breaking Changes

CWL

• Prevent simultaneous Singularity container pulls in toil-cwl-runner (#4990)
• Added support to import files on workers for toil-cwl-runner (#5025)
  • --runImportsOnWorkers to enable importing files on workers
  • --importWorkersDisk to control how much disk space the import worker will use
• Don't error when passing through input as the output (#5138)
• CWL jobs with dynamic requirements now have input type checking properly protected by their conditionals. (#4930)
• Fixed a LoadListing bug with CWL workflows (#5149)
• Fix CWL Workflow Slurm memory test (#5151)
• workDir and jobStore defaults to tmp-outdir-prefix (#5154)

WDL

• Update WDL conformance tests on CI (#4875)
• Added support to run task only WDL files (#4960)
• Added support for the gpu field in WDL (#4949)
• Support passing inputs into toil-wdl-runner for task only WDLs (#4977)
• toil-wdl-runner will now carry through task exit codes (#4978)
• toil-wdl-runner will respect explicit null values for optional inputs (#4981)
• toil-wdl-runner will not immediately error on nonexistent coerced files until outputted (#4994)
  • File? type for string to file coercion is now supported (will be nullified)
• WDL output files will now live in directories named after their tasks instead of UUID directories (#5008)
• Fixed a bug with conditional statements inside a WDL scatter (#5055)
• toil-wdl-runner now correctly finds and returns outputs from tasks in scatters and conditionals when a WDL workflow lacks an output section. (#5094)
• toil-wdl-runner has a new --allCallOutputs option to allow including all calls' outputs in a workflow's output. (#5093)
• toil-wdl-runner can now detect and try not to delete the outputs of a workflow that is meant to use the Cromwell Output Organizer (croo). Note that croo still can't actually work on the output of toil-wdl-runner. (#5093)
• --allCallOutputs no longer discards WDL workflow outputs section outputs. (#5106)
• File virtualization in toil-wdl-runner now only happens at task boundaries (#5028)
  • File to String coercion should be supported
• Added support to import files on workers for toil-wdl-runner (#5103)
• Support WDL 1.1 disk specification as per spec (#5001)
• Fixed a bug with WDL file imports (#5121)

Kubernetes

Dependencies

Misc

• Makefile: use isolated builds, add dist target (sdist+wheel) and deprecate the sdist target. (#4820) (#4826)
• Toil will now wait --jobStoreTimeout seconds (default: 30) to see an update to/removal of a job that was run, and will not let the job succeed unless it is seen to make progress. (#3814)
• Toil job descriptions no longer have a command field, and we track the link to the job body and the command to invoke the Toil worker separately. (#4811)
• Several typos in the docs were fixed (#4889)
• Add a test to ensure batchsystem plugins are installable (#4879)
• Fix Toil utils to work without the AWS extra (#4953)
• Print commit hash with toil --version when installed from source. Before: 7.1.0a1. After: 7.1.0a1-ccf57e6071e32675daabdcbacb91988e871745a9 (#4954)
• Fixed a broken URL and an omitted variable in CI tests (#4974)
• Generate default config correctly (#5014)
• Use the latest setuptools when running cactus. (#5017)
• Toil will refuse to proceed if it detects that its coordination directory or a Singularity cache directory it needs to lock is on Ceph, to prevent hanging the Ceph MDS (#4972)
• Fix a NotImplementedError in the Grid Engine batchsystem (#5061)
  • Added basic Grid Engine CI tests
• Update Cactus on CI to 2.9.0 (#5062)
• Separate out create/delete iam role functions into lib.
• Remove deprecated pipes module (#5122)
• New --slurmTime/TOIL_SLURM_TIME setting to set the time limit on Slurm jobs in a way Toil itself understands. (#5010)
• New --slurmPE argument to allow setting a parallel-job Slurm partition without using TOIL_SLURM_PE (#5010)
• New --slurmArgs argument to allow specifying extra Slurm submission arguments without using TOIL_SLURM_ARGS (#5010)
• For non-GPU jobs on Slurm, Toil will submit the job to a partition with a time limit long enough to accommodate the configured runtime (from --slurmTime). (For GPU jobs, the lowest-priority GPU partition is still always used.) (#5010)
• Toil now has a --slurmDefaultAllMem option to run jobs lacking their own memory requirements with Slurm's --mem=0, so they get a whole node's memory. (#4971)
• toil-cwl-runner now has --no-cwl-default-ram (and --cwl-default-ram) to control whether the CWL spec's default ramMin is applied, or Toil's own default memory logic is used. (#4971)
• The --dont_allocate_mem and --allocate_mem options have been deprecated and replaced with --slurmAllocateMem, which can be True or False. (#4971)
• Added WDL unit tests to CI (#5110)
• Mesos build updated. (#5049)
• CWL and WDL argument parsing revised for Python 3.12. (#5049)
• Organize stats and logging files into stats/inbox and stats/archive and avoid a circular rename. (#1727)
• Added proper FTP support for jobstores (#5134)
• URL existence and size gets/checks are now done with HEAD requests (#5134)
• Dependabot configuration should now pass schema validation and is itself under CI (#5175)
• Toil now tests a version of Cactus that ought to run on Python 3.13. (#5184)
• WDL conformance tests on Kubernetes may now run for 30 minutes. (#5185)

Thank you to our contributors: