diff --git a/bin/cylc-message b/bin/cylc-message index c4752e34a0e..54e98413201 100755 --- a/bin/cylc-message +++ b/bin/cylc-message @@ -40,7 +40,7 @@ def main(): parser.add_option( "-p", "--priority", metavar="PRIORITY", type="choice", - choices=['NORMAL', 'WARNING', 'CRITICAL'], + choices=['NORMAL', 'WARNING', 'CRITICAL', 'CUSTOM'], help="message priority: NORMAL, WARNING, or CRITICAL; default NORMAL.", action="store", dest="priority", default="NORMAL") diff --git a/doc/src/cylc-user-guide/cug.tex b/doc/src/cylc-user-guide/cug.tex index 9df11c0a326..6ca390acce4 100644 --- a/doc/src/cylc-user-guide/cug.tex +++ b/doc/src/cylc-user-guide/cug.tex @@ -5337,11 +5337,14 @@ \subsection{Inlined Tasks} a few lines of code, however, we recommend using external shell scripts to allow independent testing, re-use, and shell mode editing. -\subsection{Custom Task Messages} +\subsection{Task Messages} -Custom messages can be sent back to the suite daemon to report progress, -warnings, or critical events; and to trigger {\em warning} or {\em critical} -event handlers (see~\ref{EventHandling}). +Tasks messages can be sent back to the suite daemon to report completed +outputs and arbitrary messages of different priority levels. + +Some types of message - in addition to events like task failure - can +optionally trigger execution of event handlers in the suite daemon +(see~\ref{EventHandling}). Normal priority messages are printed to \lstinline=job.out= and logged by the suite daemon: @@ -5350,16 +5353,23 @@ \subsection{Custom Task Messages} cylc message "Hello from ${CYLC_TASK_ID}" \end{lstlisting} -Warning priority messages are printed to \lstinline=job.err=, logged by the -suite daemon, and can be passed to {\em warning} event handlers -(see~\ref{TaskEventHandling}): +CUSTOM priority messages are printed to \lstinline=job.out=, logged by the +suite daemon, and can be used to trigger {\em custom} event handlers: +\lstset{language=bash} +\begin{lstlisting} +cylc message -p CUSTOM "data available for ${CYLC_TASK_CYCLE_POINT}" +\end{lstlisting} +Task output messages, used for triggering other tasks, can also be sent with +custom priority if need be + +WARNING priority messages are printed to \lstinline=job.err=, logged by the +suite daemon, and can be passed to {\em warning} event handlers: \begin{lstlisting} cylc message -p WARNING "Uh-oh, something's not right here." \end{lstlisting} -Critical priority messages are printed to \lstinline=job.err=, logged by the -suite daemon, and can be passed to {\em critical} event handlers -(see~\ref{TaskEventHandling}): +CRITICAL priority messages are printed to \lstinline=job.err=, logged by the +suite daemon, and can be passed to {\em critical} event handlers: \begin{lstlisting} cylc message -p CRITICAL "ERROR occurred in process X!" \end{lstlisting} @@ -6600,22 +6610,21 @@ \subsection{Automatic Task Retry On Failure} the next specified delay period expires. A usage example is shown in the suite listed below under~\ref{EventHandling}. -\subsection{Suite And Task Event Handling} +\subsection{Task Event Handling} \label{EventHandling} See also~\ref{SuiteEventHandling} and~\ref{TaskEventHandling} in the {\em Suite.rc Reference}. -Cylc can call nominated event handlers when certain suite or task events occur. -This is intended to facilitate centralized alerting and automated handling of -critical events. Event handlers can be used to send a message, call a pager, -and so on; or intervene in the operation of their own suite using cylc -commands. +Cylc can call nominated event handlers - to do whatever you like - when certain +suite or task events occur. This facilitates centralized alerting and automated +handling of critical events. Event handlers can be used to send a message, call +a pager, or whatever; they can even intervene in the operation of their own +suite using cylc commands. -To send an email, you can use the built-in setting -\lstinline=[[[events]]]mail events= to specify a list of events for which -notifications should be sent. E.g.\ to send an email on (submission) failed and -retry: +To send an email, use the built-in setting \lstinline=[[[events]]]mail events= +to specify a list of events for which notifications should be sent. E.g.\ to +send an email on (submission) failed and retry: \lstset{language=suiterc} \begin{lstlisting} @@ -6647,15 +6656,14 @@ \subsection{Suite And Task Event Handling} large group of tasks all fail at similar time. See ~\ref{task-event-mail-interval} for details. -Event handler commands can be located in the suite \lstinline=bin/= directory, +Event handlers can be located in the suite \lstinline=bin/= directory; otherwise it is up to you to ensure their location is in \lstinline=$PATH= (in -the shell in which cylc runs, on the suite host). The commands should require -very little resource to run and should return quickly. (Each event -handler is invoked by a child process in a finite process pool that is also -used to submit, poll and kill jobs. The child process will wait for the event -handler to complete before moving on to the next item in the queue. If the -process pool is saturated with long running event handlers, the suite will -appear to hang.) +the shell in which the suite daemon runs). They should require little +resource and return quickly - as each event handler is invoked by a child +process in a finite process pool that is also used to submit, poll and kill +jobs. The child process will wait for the event handler to complete before +moving on to the next item in the queue. If the process pool is saturated with +long running event handlers, the suite will appear to hang. Task event handlers can be specified using the \lstinline=[[[events]]] handler= settings, where @@ -6670,6 +6678,7 @@ \subsection{Suite And Task Event Handling} \item `succeeded' - the task reported successful completion \item `warning' - the task reported a WARNING priority message \item `critical' - the task reported a CRITICAL priority message + \item `custom' - the task reported a CUSTOM priority message \item `failed' - the task failed \item `retry' - the task failed but will retry after a configured delay \item `execution timeout' - task execution timed out @@ -6683,26 +6692,19 @@ \subsection{Suite And Task Event Handling} lines or command line templates (see below) and the latter is a list of events for which these commands should be invoked. -A command line template may have any or all of these patterns which will be -substituted with actual values: -\begin{myitemize} - \item \%(event)s: event name - \item \%(suite)s: suite name - \item \%(point)s: cycle point - \item \%(name)s: task name - \item \%(submit\_num)s: submit number - \item \%(id)s: task ID (i.e.\ \%(name)s.\%(point)s) - \item \%(message)s: event message, if any -\end{myitemize} +Event handler arguments can be constructed from various templates +representing suite name; task ID, name, cycle point, message, and submit +number name; and any suite or task [meta] item. See~\ref{SuiteEventHandling} +and~\ref{TaskEventHandling} for options. -Otherwise, the command line will be called with the following command line -arguments: +If no template arguments are supplied the following default command line +will be used: \begin{lstlisting} %(event)s %(suite)s %(id)s %(message)s \end{lstlisting} -Note: The substitution patterns should not be quoted in the template strings. -Where required, the values will be quoted automatically on substitution. +{\em Note: substitution patterns should not be quoted in the template strings. +This is done automatically where required.} For an explanation of the substitution syntax, see \href{https://docs.python.org/2/library/stdtypes.html#string-formatting}{String Formatting Operations} @@ -6713,10 +6715,9 @@ \subsection{Suite And Task Event Handling} The event handler will be called as soon as the task fails, not after the retry delay period when it is resubmitted. -{\em Note that event handlers are called by cylc itself, not by the -running tasks} so if you wish to pass them additional information via -the environment you must use [cylc] \textrightarrow [[environment]], -not task runtime environments. +{\em Note that event handlers are called by the suite daemon, not by task jobs.} +If you wish to pass additional information to them use [cylc] \textrightarrow +[[environment]], not task runtime environment. The following 2 \lstinline=suite.rc= snippets are examples on how to specify event handlers using the alternate methods: @@ -6742,8 +6743,8 @@ \subsection{Suite And Task Event Handling} handlers = "echo '!!!!!EVENT!!!!!' " handler events = retry, failed \end{lstlisting} - -Note: The handler command is called like this: +The handler command here - specified with no arguments - is called with the +default arguments, like this: \begin{lstlisting} echo '!!!!!EVENT!!!!!' %(event)s %(suite)s %(id)s %(message)s \end{lstlisting} diff --git a/doc/src/cylc-user-guide/suiterc.tex b/doc/src/cylc-user-guide/suiterc.tex index b2a08d5ddb5..48299f0ad0d 100644 --- a/doc/src/cylc-user-guide/suiterc.tex +++ b/doc/src/cylc-user-guide/suiterc.tex @@ -340,22 +340,21 @@ \subsection{[cylc]} \item \%(suite)s: suite name \item \%(suite\_url)s: suite URL \item \%(message)s: event message, if any + \item any suite [meta] item, e.g.: + \begin{myitemize} + \item \%(title)s: suite title + \item \%(importance)s: example custom suite metadata + \end{myitemize} \end{myitemize} -It can also include the patterns from the [meta] section of the suite, for example: -\begin{myitemize} - \item \%(suite-priority)s: suite priority - \item \%(color)s: suite color -\end{myitemize} - -Otherwise, the command line will be called with the following command line +Otherwise the command line will be called with the following default arguments: \begin{lstlisting} %(event)s %(suite)s %(message)s \end{lstlisting} -Note: The substitution patterns should not be quoted in the template strings. -Where required, the values will be quoted automatically on substitution. +{\em Note: substitution patterns should not be quoted in the template strings. +This is done automatically where required.} Additional information can be passed to event handlers via [cylc] \textrightarrow [[environment]]. @@ -1612,50 +1611,47 @@ \subsection{[runtime]} \label{TaskEventHandling} Cylc can call nominated event handlers when certain task events occur. This -section configures specific task event handlers; see~\ref{SuiteEventHandling} for -suite event hooks. +section configures specific task event handlers; see~\ref{SuiteEventHandling} +for suite events. -Event handler commands can be located in the suite \lstinline=bin/= directory, +Event handlers can be located in the suite \lstinline=bin/= directory, otherwise it is up to you to ensure their location is in \lstinline=$PATH= (in -the shell in which cylc runs, on the suite host). The commands should require -very little resource to run and should return quickly. +the shell in which the suite daemon runs). They should require little resource +to run and return quickly. Each task event handler can be specified as a list of command lines or command -line templates. - -A command line template may have any or all of these patterns which will be -substituted with actual values: +line templates. They can contain any or all of the following patterns, which +will be substituted with actual values: \begin{myitemize} \item \%(event)s: event name \item \%(suite)s: suite name \item \%(point)s: cycle point \item \%(name)s: task name - \item \%(suite\_url)s: suite URL - \item \%(task\_url)s: task URL \item \%(submit\_num)s: submit number \item \%(id)s: task ID (i.e.\ \%(name)s.\%(point)s) \item \%(message)s: event message, if any + \item any task [meta] item, e.g.: + \begin{myitemize} + \item \%(title)s: task title + \item \%(URL)s: task URL + \item \%(importance)s - example custom task metadata + \end{myitemize} + \item any suite [meta] item, prefixed with ``suite\_'', e.g.: + \begin{myitemize} + \item \%(suite\_title)s: suite title + \item \%(suite\_URL)s: suite URL + \item \%(suite\_rating)s - example custom suite metadata + \end{myitemize} \end{myitemize} -The template can also include the patterns from [[[meta]]] section of the task, for example: -\begin{myitemize} - \item \%(importance)s: task priority - \item \%(color)s: task color -\end{myitemize} - -Patterns from the [meta] section of the suite can also be used, for example: -\begin{myitemize} - \item \%(suite\_priority)s: suite priority - \item \%(suite\_color)s: suite color -\end{myitemize} - -Otherwise, the command line will be called with the following arguments: +Otherwise, the command line will be called with the following default +arguments: \begin{lstlisting} %(event)s %(suite)s %(id)s %(message)s \end{lstlisting} -Note: The substitution patterns should not be quoted in the template strings. -Where required, the values will be quoted automatically on substitution. +{\em Note: substitution patterns should not be quoted in the template strings. +This is done automatically where required.} For an explanation of the substitution syntax, see String Formatting Operations in the Python documentation: @@ -1684,6 +1680,7 @@ \subsection{[runtime]} \item {\bf execution timeout} - the task timed out after execution commenced \item {\bf warning} - the task reported a WARNING priority message \item {\bf critical} - the task reported a CRITICAL priority message + \item {\bf custom} - the task reported a CUSTOM priority message \end{myitemize} Item details: diff --git a/lib/cylc/cfgspec/suite.py b/lib/cylc/cfgspec/suite.py index d934eaa3bdf..fc558352c26 100644 --- a/lib/cylc/cfgspec/suite.py +++ b/lib/cylc/cfgspec/suite.py @@ -409,6 +409,7 @@ def _coerce_parameter_list(value, keys, _): 'submission retry handler': vdr(vtype='string_list'), 'execution timeout handler': vdr(vtype='string_list'), 'submission timeout handler': vdr(vtype='string_list'), + 'custom handler': vdr(vtype='string_list'), }, 'suite state polling': { 'user': vdr(vtype='string'), diff --git a/lib/cylc/task_events_mgr.py b/lib/cylc/task_events_mgr.py index 0a61bc318aa..dd5c2fbf4c2 100644 --- a/lib/cylc/task_events_mgr.py +++ b/lib/cylc/task_events_mgr.py @@ -385,8 +385,9 @@ def process_message(self, itask, priority, message, poll_event_time=None, priority = getLevelName(priority) self._db_events_insert( itask, ("message %s" % str(priority).lower()), message) - - if priority in [TaskMessage.WARNING, TaskMessage.CRITICAL]: + if priority == "CUSTOM": + self.setup_event_handlers(itask, "custom", message) + elif priority in [TaskMessage.WARNING, TaskMessage.CRITICAL]: self.setup_event_handlers(itask, priority.lower(), message) def setup_event_handlers(self, itask, event, message): diff --git a/lib/cylc/task_message.py b/lib/cylc/task_message.py index 30172f6ec6e..fbc155f89a4 100644 --- a/lib/cylc/task_message.py +++ b/lib/cylc/task_message.py @@ -46,7 +46,8 @@ class TaskMessage(object): NORMAL = "NORMAL" WARNING = "WARNING" CRITICAL = "CRITICAL" - PRIORITIES = (NORMAL, WARNING, CRITICAL) + CUSTOM = "CUSTOM" + PRIORITIES = (NORMAL, WARNING, CRITICAL, CUSTOM) MSG_RETRY_INTVL = 5.0 MSG_MAX_TRIES = 7 @@ -91,7 +92,7 @@ def send(self, messages): def _print_messages(self, messages): """Print message to send.""" - if self.priority == self.NORMAL: + if self.priority in [self.NORMAL, self.CUSTOM]: handle = sys.stdout else: handle = sys.stderr diff --git a/tests/cylc-get-config/00-simple/section2.stdout b/tests/cylc-get-config/00-simple/section2.stdout index 7e886c28949..aba9d9e5a9e 100644 --- a/tests/cylc-get-config/00-simple/section2.stdout +++ b/tests/cylc-get-config/00-simple/section2.stdout @@ -23,6 +23,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -100,6 +101,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -176,6 +178,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -253,6 +256,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -330,6 +334,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -418,6 +423,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -495,6 +501,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -572,6 +579,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -648,6 +656,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -725,6 +734,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -802,6 +812,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -878,6 +889,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = @@ -955,6 +967,7 @@ submission retry handler = warning handler = critical handler = + custom handler = succeeded handler = retry handler = failed handler = diff --git a/tests/events/38-task-event-handler-custom.t b/tests/events/38-task-event-handler-custom.t new file mode 100755 index 00000000000..34600b78a62 --- /dev/null +++ b/tests/events/38-task-event-handler-custom.t @@ -0,0 +1,34 @@ +#!/bin/bash +# THIS FILE IS PART OF THE CYLC SUITE ENGINE. +# Copyright (C) 2008-2017 NIWA +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Test custom priority event handling. +. "$(dirname "$0")/test_header" +set_test_number 6 +install_suite "${TEST_NAME_BASE}" "${TEST_NAME_BASE}" + +run_ok "${TEST_NAME_BASE}-validate" cylc validate "${SUITE_NAME}" +suite_run_ok "${TEST_NAME_BASE}-run" \ + cylc run --debug --reference-test "${SUITE_NAME}" +FOO_ACTIVITY_LOG="${SUITE_RUN_DIR}/log/job/1/foo/NN/job-activity.log" +SUITE_LOG="${SUITE_RUN_DIR}/log/suite/log" +grep_ok \ +"\[(('event-handler-00', 'custom'), 1) out\] !!CUSTOM!! foo.1 fugu Data ready for barring" \ + "${FOO_ACTIVITY_LOG}" +grep_ok "\[foo.1\].*Data ready for barring" $SUITE_LOG +grep_ok "\[foo.1\].*Data ready for bazzing" $SUITE_LOG +grep_ok "\[foo.1\].*Aren't the hydrangeas nice?" $SUITE_LOG +purge_suite "${SUITE_NAME}" diff --git a/tests/events/38-task-event-handler-custom/reference.log b/tests/events/38-task-event-handler-custom/reference.log new file mode 100644 index 00000000000..d0735db04f4 --- /dev/null +++ b/tests/events/38-task-event-handler-custom/reference.log @@ -0,0 +1,5 @@ +2017-08-26T13:08:15+12 INFO - Initial point: 1 +2017-08-26T13:08:15+12 INFO - Final point: 1 +2017-08-26T13:08:15+12 INFO - [foo.1] -triggered off [] +2017-08-26T13:08:19+12 INFO - [bar.1] -triggered off ['foo.1'] +2017-08-26T13:08:24+12 INFO - [baz.1] -triggered off ['foo.1'] diff --git a/tests/events/38-task-event-handler-custom/suite.rc b/tests/events/38-task-event-handler-custom/suite.rc new file mode 100644 index 00000000000..a9c95b76642 --- /dev/null +++ b/tests/events/38-task-event-handler-custom/suite.rc @@ -0,0 +1,31 @@ +[cylc] + [[reference test]] + required run mode = live + live mode suite timeout = PT30S +[scheduling] + [[dependencies]] + graph = """foo:a => bar + foo:b => baz""" +[runtime] + [[root]] + script = sleep 1 + [[[events]]] + custom handler = echo !!CUSTOM!! %(name)s.%(point)s %(fish)s %(message)s + [[foo]] + script = """ +sleep 2 +# Output message for triggering, and custom event handler. +cylc message -p CUSTOM "Data ready for barring" +sleep 2 +# Generic message, not for triggering or custom event handler. +cylc message "Aren't the hydrangeas nice?" +sleep 2 +# Output message for triggering, not custom event handler. +cylc message "Data ready for bazzing" +sleep 2 +""" + [[[outputs]]] + a = "Data ready for barring" + b = "Data ready for bazzing" + [[[meta]]] + fish = fugu