Controlled Commit States

[sfmskywalker]

Objective:
To enhance the current workflow execution process by introducing the concept of "commit points". This will allow for more frequent and controlled persistence of workflow state to the database, beyond the existing default behaviour of persisting only upon completion or when blocked.

Background:
Currently, workflows are executed in-memory and are persisted to the configured data store only upon completion or in scenarios where they can no longer proceed (e.g., due to blocking activities). This approach, while efficient, lacks the flexibility to capture interim states of the workflow during its execution, potentially leading to data loss or inconsistency in case of unexpected interruptions.

Requirements:

1. Configurable Commit Points for Workflows:

   • Introduce a setting at the workflow level to specify commit behaviour. This includes:
     • Persisting the workflow state to the database before execution begins.
     • Persisting after every activity within the workflow.
     • Persisting after activities designated as "commit points".

2. Commit Settings for Individual Activities:

   • Implement an option for each activity within a workflow where users can specify:
     • Committing the workflow state to the database before the execution of the activity.
     • Committing after the activity has been executed.

Expected Benefits:
This feature will provide users with greater control over the persistence of workflow states, enhancing reliability and data integrity. It allows for recovery of workflows to a more recent state in case of interruptions, thereby reducing the risk of data loss and improving overall system resilience.