List of notable changes, for a complete list of changes see the closed milestones for each release.
#6130 - Prevent commands accepting job IDs where it doesn't make sense.
#6170 - Fix an issue where the Cylc logo could appear in the workflow log.
#6176 - Fix bug where jobs which fail to submit are not shown in GUI/TUI if submission retries are set.
#5600 - The cylc dump command now only shows active tasks (e.g. running & queued
tasks). This restores its behaviour of only showing the tasks which currently
exist in the pool as it did in Cylc 7 and earlier versions of Cylc 8.
#5727 - Cylc now ignores PYTHONPATH to make it more robust to task environments which set this value. If you want to add to the Cylc environment itself, e.g. to install a Cylc extension, use CYLC_PYTHONPATH.
#5794 - Remove cylc report-timings from automatic installation with pip install cylc-flow[all]. If you now wish to install it use pip install cylc-flow[report-timings]. cylc report-timings is incompatible with Python 3.12.
#5836 - Removed the 'CYLC_TASK_DEPENDENCIES' environment variable
#5956 - cylc lint: deprecated [cylc-lint] section in favour of [tool.cylc.lint] in pyproject.toml
#6046 - The submit-fail and expire task outputs must now be
optional
and can no longer be
required.
#5571 - Make workflow CYLC_ variables available to the template processor during parsing.
#5658 - New "cylc set" command for setting task prerequisites and outputs.
#5709 - Forward arbitrary environment variables over SSH connections
#5721 - Allow task simulation mode settings to be changed dynamically using cylc broadcast.
#5731 - Major upgrade to cylc tui which now supports larger workflows and can browse installed workflows.
#5738 - Optionally spawn parentless xtriggered tasks sequentially - i.e., one at a time, after the previous xtrigger is satisfied, instead of all at once out to the runahead limit. The wall_clock xtrigger is now sequential by default.
#5769 - Include task messages and workflow port as appropriate in emails configured by "mail events".
#5803 - Updated 'reinstall' functionality to support multiple workflows
#5809 - The workflow-state command and xtrigger are now flow-aware and take universal IDs instead of separate arguments for cycle point, task name, etc. (which are still supported, but deprecated).
#5831 - Add capability to install xtriggers via a new cylc.xtriggers entry point
#5864 - Reimplemented the suite-state xtrigger for interoperability with Cylc 7.
#5872 - Improvements to cylc clean remote timeout handling.
#5873 - cylc lint improvements:
- Allow use of
#noqa: S001comments to skip checks for a single line. - Stop
cylc lintobjecting to%include <file>syntax.
#5879 - cylc lint now warns of use of old templated items such as %(suite)s
#5890 - Lint: Warn users that setting CYLC_VERSION, ROSE_VERSION or
FCM_VERSION in the workflow config is deprecated.
#5943 - The stop after cycle point can now be specified as an offset from the inital cycle point.
#5955 - Support xtrigger argument validation.
#6029 - Workflow graph window extent is now preserved on reload.
#6046 - The condition that Cylc uses to evaluate task output completion can now be
customized in the [runtime] section with the new completion configuration.
This provides a more advanced way to check that tasks generate their required
outputs when run.
#5809 - Fix bug where the "cylc workflow-state" command only polled for task-specific status queries and custom outputs.
#6008 - Fixed bug where the [scheduler][mail]to/from settings did not apply as defaults for task event mail.
#6036 - Fixed bug in simulation mode where repeated submissions were not displaying correctly in TUI/GUI.
#6067 - Fixed a bug that sometimes allowed suicide-triggered or manually removed tasks to be added back later.
#6109 - Fixed bug affecting job submission where the list of bad hosts was not always reset correctly.
#6123 - Allow long-format datetime cycle points in IDs used on the command line.
#6096 - Fixed bug that caused graph arrows to go missing in the GUI when suicide triggers are present.
#6102 - Fixed bug introduced in 8.2.6 in cylc vip & cylc vr when using cylc-rose options (-S, -D, -O).
#6068 - Removed the Rose Options (-S, -O, -D) from cylc play. If you need these use them with cylc install.
#6072 - Nano Syntax Highlighting now available.
#6071 - cylc config now shows xtrigger function signatures.
#6078 - Fixed bug where cylc lint could hang when checking inherit settings in flow.cylc.
#5924 - Validation: a cycle offset can only appear on the right of a dependency if the task's cycling is defined elsewhere with no offset.
#5933 - Fixed bug in cylc broadcast (and the GUI Edit Runtime command) where everything after a # character in a setting would be stripped out.
#5959 - Fix an issue where workflow "timeout" events were not fired in all situations when they should have been.
#6011 - Fixed a cylc vip bug causing remote re-invocation to fail if using --workflow-name option.
#6031 - Fixed workflow-state command and xtrigger for alternate cylc-run directory.
#5772 - cylc lint: added a check for indentation being 4N spaces.
#5838 - cylc lint: added rule to check for rose date usage (should be replaced with isodatetime).
#5789 - Prevent the run mode from being changed on restart.
#5801 - Fix traceback when using parentheses on right hand side of graph trigger.
#5821 - Fixed issue where large uncommitted changes could cause cylc install to hang.
#5841 - cylc lint: improved handling of S011 to not warn if the # is #$ (e.g. shell base arithmetic).
#5885 - Fixed bug in using a final cycle point with chained offsets e.g. 'final cycle point = +PT6H+PT1S'.
#5893 - Fixed bug in computing a time interval-based runahead limit when future triggers are present.
#5902 - Fixed a bug that prevented unsetting execution time limit by broadcast or reload.
#5908 - Fixed bug causing redundant DB updates when many tasks depend on the same xtrigger.
#5909 - Fix a bug where Cylc VIP did not remove --workflow-name= from Cylc play arguments.
#5660 - Re-worked graph n-window algorithm for better efficiency.
#5753 - Fixed bug where execution time limit polling intervals could end up incorrectly applied
#5776 - Ensure that submit-failed tasks are marked as incomplete (so remain visible) when running in back-compat mode.
#5791 - fix a bug where if multiple clock triggers are set for a task only one was being satisfied.
#5237 - Back-compat: allow workflow-state xtriggers (and the cylc workflow-state
command) to read Cylc 7 databases.
#5693 - Log command issuer, if not the workflow owner, for all commands.
#5694 - Don't fail config file parsing if current working directory does not exist. (Note however this may not be enough to prevent file parsing commands failing elsewhere in the Python library).
#5704 - Fix off-by-one error in automatic upgrade of Cylc 7 "max active cycle points" to Cylc 8 "runahead limit".
#5708 - Fix runahead limit at start-up, with recurrences that start beyond the limit.
#5755 - Fixes an issue where submit-failed tasks could be incorrectly considered as completed rather than causing the workflow to stall.
#5631 - Fix bug in remote clean for workflows that generated flow.cylc files at runtime.
#5650 - Fix a bug preventing clean-up of finished tasks in the GUI and TUI.
#5685 - Fix "cylc pause" command help (it targets workflows, not tasks, but was printing task-matching documentation as well).
#5600 -
The CYLC_TASK_DEPENDENCIES environment variable will no longer be exported
in job environments if there are more than 50 dependencies. This avoids an
issue which could cause jobs to fail if this variable became too long.
#5992 - Before trying to reload the workflow definition, the scheduler will now wait for preparing tasks to submit, and pause the workflow. After successful reload the scheduler will unpause the workflow.
#5605 - Added -z shorthand
option for defining a list of strings:
- Before:
cylc command -s "X=['a', 'bc', 'd']" - After:
cylc command -z X=a,bc,d.
#5537 - Allow parameters
in family names to be split, e.g. <foo>FAM<bar>.
#5589 - Move to workflow directory during file parsing, to give the template processor access to workflow files.
#5405 - Improve scan command help, and add scheduler PID to the output.
#5461 - preserve colour
formatting when starting workflows in distributed mode using run hosts.
#5291 - re-implement old-style clock triggers as wall_clock xtriggers.
#5439 - Small CLI short option chages:
Add the -n short option for --workflow-name to cylc vip; rename the -n
short option for --no-detach to -N; add -r as a short option for
--run-name.
#5231 - stay up for a timeout period on restarting a completed workflow, to allow for manual triggering.
#5549,
#5546 -
Various enhancements to cylc lint:
cylc lintwill provide a non-zero return code if any issues are identified. This can be overridden using the new--exit-zeroflag.- Fix numbering of lint codes (n.b. lint codes should now be permenantly
unchanging, but may have changed since Cylc 8.1.4, so
pyproject.tomlfiles may need updating). - Check for suicide triggers in
.cylcfiles. - Check for
platform = $(rose host-select). - Check for use of deprecated Cylc commands (and
rose suite-hook). - Check for zero prefixed Jinja2 integers.
- Only check for missing Jinja2 shebangs in
flow.cylcandsuite.rcfiles.
#5525 - Jobs can use scripts
in share/bin and Python modules in share/lib/python.
#5328 - Efficiency improvements to reduce task management overheads on the Scheduler.
#5611 - Improve the documentation of the GraphQL schema.
#5616 - Improve PBS support for job IDs with trailing components.
#5619 -
Fix an issue where the task_pool table in the database wasn't being updated
in a timely fashion when tasks completed.
#5606 - Task outputs and messages are now validated to avoid conflicts with built-in outputs, messages, qualifiers and Cylc keywords.
#5614 -
Fix a bug in Cylc 7 compatibility mode where tasks running in the none flow
(e.g. via cylc trigger --flow=none) would trigger downstream tasks.
#5604 -
Fix a possible issue where workflows started using
cylc play --start-cycle-point could hang during startup.
#5573 - Fix bug that ran a
queued waiting task even after removal by cylc remove.
#5524 - Logging includes timestamps
for cylc play when called by cylc vip or cylc vr.
#5228 - Enabled the "stop", "poll", "kill" and "message" commands to be issued from the UI whilst the workflow is in the process of shutting down.
#5582 - Set Cylc 7 compatibility mode before running pre-configure plugins.
#5587 - Permit commas in xtrigger arguments and fix minor issues with the parsing of xtrigger function signatures.
#5618 - Fix a bug when rapidly issuing the same/opposite commands e.g. pausing & resuming a workflow.
#5625 - Exclude setuptools
version (v67) which results in dependency check failure with editable installs.
#5514 -
Ensure cylc cat-log directory listings always include the job-activity.log
file when present and are able to list submit-failed jobs.
#5506 -
Fix bug introduced in 8.1.3 where specifying a subshell command for
flow.cylc[runtime][<namespace>][remote]host (e.g. $(rose host-select))
would always result in localhost.
#5475 - much faster computation of the visualization window around active tasks (at the cost, for now, of not showing non-active "cousin" nodes).
#5453 - cylc cat-log can now
list and view workflow log files including install logs and workflow
configuration files.
#5495 - Fix bug that could cause invalid parent tasks to appear in the UI datastore.
#5334 - Apply graph prerequisite changes to already-spawned tasks after reload or restart.
5466 - Don't generate duplicate prerequisites from recurrences with coincident points.
5450 - Validation provides better error messages if [sections] and settings are mixed up in a configuration.
5445 - Fix remote tidy bug where install target is not explicit in platform definition.
5398 - Fix platform from group selection order bug.
#5395 - Fix bug where workflow shuts down if all hosts for all platforms in a platform group are unreachable.
#5384 -
Fixes cylc set-verbosity.
#5479 -
Fixes cylc help license
#5394 - Fixes a possible scheduler traceback observed with remote task polling.
#5386 - Fix bug where
absence of job name length maximum in PBS platform settings would cause
Cylc to crash when preparing the job script.
#5343 - Fix a bug causing platform names to be checked as if they were hosts.
#5359 - Fix bug where viewing
a workflow's log in the GUI or using cylc cat-log would prevent cylc clean
from working.
#5349 - Bugfix: cylc vip --workflow-name
only worked when used with a space, not an =.
#5367 - Enable using
Rose options (-O, -S & -D) with cylc view.
#5363 Improvements and bugfixes
for cylc lint.
#5313 - Fix a bug causing Cylc to be unable to parse previously played Cylc 7 workflows.
#5312 - task names must be comma-separated in queue member lists. Any implicit tasks (i.e. with no task definition under runtime) assigned to a queue will generate a warning.
#5314 - Fix broken
command option: cylc vip --run-name.
#5319,
#5321,
#5325 -
Various efficiency optimisations to the scheduler which particularly impact
workflows with many-to-many dependencies (e.g. <a> => <b>).
- Workflows started with Cylc 8.0 which contain multiple "flows" cannot be restarted with Cylc 8.1 due to database changes.
#5229 -
- Added a single command to validate a previously run workflow against changes to its source and reinstall a workflow.
- Allows Cylc commands (including validate, list, view, config, and graph) to load template variables
configured by
cylc installandcylc play.
#5121 - Added a single command to validate, install and play a workflow.
#5184 - Scan for active runs of the same workflow at install time.
#5084 - Assign the most recent previous flow numbers to tasks triggered when no flows are present (e.g. on restarting a finished workflow).
#5032 - Set a default limit of 100 for the "default" queue.
#5055 and
#5086 - Upgrades to cylc lint
- Allow users to ignore Cylc Lint issues using
--ignore <Issue Code>. - Allow settings for
cylc lintto be recorded in a pyproject.toml file. - Allow files to be excluded from
cylc lintchecks.
#5081 - Reduced amount that gets logged at "INFO" level in scheduler logs.
#5259 - Add flow_nums to task_jobs table in the workflow database.
#5286 - Fix bug where
[scheduling][special tasks]clock-trigger would skip execution retry delays.
#5292 - Fix an issue where polling could be repeated if the job's platform was not available.
Maintenance release.
##5205 - Fix bug which caused orphaned running tasks to silently skip remote file installation at scheduler restart.
#5224 - workflow installation: disallow reserved names only in the top level source directory.
#5211 - Provide better
explanation of failure if icp = next (T-02, T-32) when list should be
semicolon separated.
#5196 - Replace traceback with warning, for scan errors where workflow is stopped.
#5199 - Fix a problem with the consolidation tutorial.
#5195 - Fix issue where workflows can fail to shutdown due to unavailable remote platforms and make job log retrieval more robust.
Maintenance release.
#5192 -
Recompute runahead limit after use of cylc remove.
#5188 -
Fix task state selectors in cylc trigger and other commands.
#5125 - Allow rose-suite.conf
changes to be considered by cylc reinstall.
#5023, #5187 - tasks force-triggered after a shutdown was ordered should submit to run immediately on restart.
#5137 -
Install the ana/ directory to remote platforms by default.
#5146 - no-flow tasks should not retrigger incomplete children.
#5104 - Fix retriggering of failed tasks after a reload.
#5139 - Fix bug where
cylc install could hang if there was a large uncommitted diff in the
source dir (for git/svn repos).
#5131 - Infer workflow run number
for workflow_state xtrigger.
Maintenance release.
#5115 - Updates rsync commands to make them compatible with latest rsync releases.
#5119 - Fix formatting of deprecation warnings at validation.
#5067 - Datastore fix for taskdefs removed before restart.
#5066 - Fix bug where
.cylcignore only found if cylc install is run in source directory.
#5091 - Fix problems with tutorial workflows.
#5098 - Fix bug where final task status updates were not being sent to UI before shutdown.
#5114 - Fix bug where validation errors during workflow startup were not printed to stderr before daemonisation.
#5110 - Fix bug where reloading a stalled workflow would cause it stall again.
Maintenance release.
#5025 - Fix a bug where polling causes a failed task to be shown as submitted when the workflow is reloaded.
#5045 - Fix issue where unsatisfied xtriggers could be wiped on reload.
#5031 - Fix bug where
specifying multiple datetime offsets (e.g. final cycle point = +P1M-P1D)
would not obey the given order.
#5033 - Running cylc clean
on a top level dir containing run dir(s) will now remove that top level dir
in addition to the run(s) (if there is nothing else inside it).
#5007 - Fix for cylc broadcast
cycle point validation in the UI.
#5037 - Fix bug where the workflow restart number would get wiped on reload.
#5049 - Fix several small bugs related to auto restart.
#5062 - Fix bug where preparing tasks could sometimes get orphaned when an auto restart occurred.
Cylc 8 production-ready release.
- Python 2 -> 3.
- Internal communications converted from HTTPS to ZMQ (TCP).
- PyGTK GUIs replaced by:
- Terminal user interface (TUI) included in cylc-flow.
- Web user interface provided by the cylc-uiserver package.
- A new scheduling algorithm with support for branched workflows.
- Command line changes:
cylc run->cylc playcylc restart->cylc playrose suite-run->cylc install; cylc play <id>
- The core package containing Cylc scheduler program has been renamed cylc-flow.
- Cylc review has been removed, the Cylc 7 version remains Cylc 8 compatible.
- New documentation.
See the migration guide for a full list of changes.
#4964 -
cylc reinstall now displays the changes it would make when run
interactively and has improved help / documentaiton.
#4836 - The log directory has
been tidied. Workflow logs are now found in log/scheduler rather than
log/workflow, filenames now include start/restart. Other minor directory
changes. Remote file installation logs are now per install target.
#4938 - Detect bad Platforms config: background and at job runners should have a single host.
#4877 - Upgrade the version of Jinja2 used by Cylc from 2.11 to 3.0.
#4896 - Allow the setting of default job runner directives for platforms.
#4900 - Added a command to assist
with upgrading Cylc 7 workflows to Cylc 8: Try cylc lint <workflow-dir>.
#5009 - Added new job
environment variable $CYLC_WORKFLOW_NAME_BASE as the basename of
$CYLC_WORKFLOW_NAME.
#4993 - Remove the few remaining
uses of a configured text editor (via cylc view and cylc cat-log options).
The primary uses of it (cylc trigger --edit and cylc edit in Cylc 7) have
already been removed from Cylc 8.
#5011 - Removes preparing jobs appearing in UI, and reuse submit number on restart for preparing tasks.
#5008 - Autospawn absolute-triggered tasks exactly the same way as parentless tasks.
#4984 -
Fixes an issue with cylc reload which could cause preparing tasks to become
stuck.
#4976 - Fix bug causing tasks to be stuck in UI due to discontinued graph of optional outputs.
#4975 - Fix selection of
platforms from [job] and [remote] configs.
#4948 - Fix lack of
errors/warnings for deprecated [runtime][<task>][remote]retrieve job logs *
settings.
#4970 - Fix handling of suicide triggers in back-compat mode.
#4887 - Disallow relative paths
in global.cylc[install]source dirs.
- Fix delayed spawning of parentless tasks that do have parents in a previous cycle point.
- Make integer-interval runahead limits consistent with time-interval limits:
P0means just the runahead base point;P1the base point and the point (i.e. one cycle interval), and so on.
#4936 - Fix incorrect error messages when workflow CLI commands fail.
#4941 - Fix job state for platform submit-failures.
#4931 - Fix cylc install for installing workflows from multi-level directories.
#4926 - Fix a docstring formatting problem presenting in the UI mutation flow argument info.
#4891 - Fix bug that could cause past jobs to be omitted in the UI.
#4860 - Workflow validation now fails if owner setting is used, as that setting no longer has any effect.
#4978 - cylc clean: fix
occasional failure to clean on remote hosts due to leftover contact file.
#4889 - cylc clean: don't
prompt if no matching workflows.
#4890 - cylc install: don't
overwrite symlink dir targets if they were not cleaned properly before.
#4881 - Fix bug where commands targeting a specific cycle point would not work if using an abbreviated cycle point format.