CronWorkflow gets scheduled on schedule change if new schedule is between current time and last schedule.

[gpanagiotidis]

Summary

I have noticed that CronWorkflows get executed even when they are not scheduled to do so. This happens if you change the schedule of the cronworkflow and put it in another time, that is after the time of the previous schedule, but before current time.

I think that the workflow controller checks the new schedule, and sees that a cronworkflow should have been executed in the meantime, triggering it.

What is more strange though is that argo cron list doesn't capture this behavior, so pods are being scheduled, but you can not find through argo.

Diagnostics

I am using:
Argo Workflows version 3.2.0
Kubernetes version 1.21.1
Executor: PNS

To replicate. Let's say it is 13:50 and we want to schedule a CronWorkflow that executes at 14:00 each day:

Create this CronWorkflow:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test
spec:
  schedule: "0 14 * * *"
  suspend: false
  concurrencyPolicy: "Forbid"
  failedJobsHistoryLimit: 1
  successfulJobsHistoryLimit: 1
  timezone: CET
  workflowSpec:
    entrypoint: job
    templates:
    - name: job
      container:
        image: centos:centos7
        command: ["/bin/sh","-c"]
        args: ["/bin/echo \"hello argo\""]
        imagePullPolicy: Always

A Workflow will be created at 14:00 correctly. However, let's say the time now is 14:10 and we decide that for the next days, we want the schedule to be at 14:05 instead of 14:00.

Change the cronworkflow to:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test
spec:
  schedule: "5 14 * * *"
  suspend: false
  concurrencyPolicy: "Forbid"
  failedJobsHistoryLimit: 1
  successfulJobsHistoryLimit: 1
  timezone: CET
  workflowSpec:
    entrypoint: job
    templates:
    - name: job
      container:
        image: centos:centos7
        command: ["/bin/sh","-c"]
        args: ["/bin/echo \"hello argo\""]
        imagePullPolicy: Always

and do kubectl apply -f test.yaml

This will trigger a new workflow even though the time now is 14:10 (5 minutes after the supposed new schedule).

However using argo cron list, in the LAST RUN field we get workflow that was execute at 14:00 and not at the 14:05.

So two things here.

1. Is it the expected behavior to execute a cronworkflow outside of the schedule like this?
2. Shouldn't the LAST RUN show the correct last executed workflow?

The expected behavior for me would be to execute the CronWorkflow on the schedule.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[simster7]

I'll take a look

[gpanagiotidis]

I think that the controller behaves as if the workflow was missed, and since no startingDeadlingSeconds is set, it gets executed immediately.

[simster7]

Yup, that's my thought as well

[Pluggi]

I have the same issue with 3.1.1.

Using kubectl replace does not trigger the issue.

[sarabala1979]

@simster7 can you please look into this issue?

[simster7]

Yes, I'll look

[simster7]

I took a look.

I think that the controller behaves as if the workflow was missed, and since no startingDeadlingSeconds is set, it gets executed immediately.

This is correct. When you edit a cron workflow and it gets reloaded by the workflow controller, it has no way of knowing if a change was made to the Schedule or if an execution was missed. Using replace works because it replaces the whole cron workflow (including the Status fields which contain the last known execution time).

Two ways forward:

• As a workaround, simply use kubectl replace to modify cron workflows. Improve UX with additional argo cron command, argo cron replace #7344. Also, if using kubectl edit or similar, simply delete the status.lastScheduledTime field.
• For best UX, record in CronWorkflow.Status a field called lastAppliedSchedule and if the current schedule does not match lastAppliedSchedule, then do not execute missed workflows
  @gpanagiotidis