[BUG] Exception when saving workflow execution logs after running alterations when using MongoDb storage

[rosca-sabina]

Description

When using MongoDB as storage and the default workflow runtime and default workflow execution pipeline, an exception is thrown and recorded as an incident on the WorkflowExecutionContext when running an alteration on an existing workflow instance. The exception's stack trace is:

System.ArgumentException: Must contain at least 1 request. (Parameter 'requests')
    at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
    at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
    at Elsa.MongoDb.Common.MongoDbStore`1.AddManyAsync(IEnumerable`1 documents, CancellationToken cancellationToken)
    at Elsa.Workflows.Runtime.Services.StoreWorkflowExecutionLogSink.PersistExecutionLogsAsync(WorkflowExecutionContext context, CancellationToken cancellationToken)
    at Elsa.Workflows.Runtime.Middleware.Workflows.PersistWorkflowExecutionLogMiddleware.InvokeAsync(WorkflowExecutionContext context)
    at Elsa.Workflows.Runtime.Middleware.Workflows.PersistActivityExecutionLogMiddleware.InvokeAsync(WorkflowExecutionContext context)
    at Elsa.Workflows.Runtime.Middleware.Workflows.PersistBookmarkMiddleware.InvokeAsync(WorkflowExecutionContext context)
    at Elsa.Workflows.Runtime.Middleware.Workflows.ScheduleBackgroundActivitiesMiddleware.InvokeAsync(WorkflowExecutionContext context)
    at Elsa.Workflows.Middleware.Workflows.EngineExceptionHandlingMiddleware.InvokeAsync(WorkflowExecutionContext context)

Steps to Reproduce

To help us identify the issue more quickly, please follow these guidelines:

1. Detailed Steps:

   • Use Elsa Workflows version 3.2.0-rc4 with the default workflow runtime, default workflow execution pipeline and MongoDB as storage for workflow runtime and management.
   • Run a workflow using IWorkflowRuntime.StartWorkflowAsync.StartWorkflowAsync(string definitionId, StartWorkflowRuntimeParams? options).
   • Run any sort of alteration on the workflow using IAlterationRunner.RunAsync(string workflowInstanceId, IEnumerable<IAlteration> alterations, CancellationToken cancellationToken).
   • After running the alteration, export the workflow instance's state using IWorkflowRuntime.ExportWorkflowStateAsync(string workflowInstanceId, CancellationToken cancellationToken). The workflow state's Incidents property will contain an incident with the exception I described under description, even if the alteration was run successfully (denoted by RunAlterationsResult.IsSuccessful being true).

2. Attachments:

   • Sample Project: I created a console application that reproduces the issue: ElsaAlterationsBugRepro.zip. The example requires a MongoDB instance running on localhost:27017 to run.

3. Reproduction Rate: The issue appears every time in the given conditions.

Expected Behavior

I expect to be able to run alterations when using MongoDB as storage without having incidents on the workflow instance.

Actual Behavior

When using MongoDB as storage, running alterations on a workflow instance causes

Environment

• Elsa Package Version: 3.2.0-rc4
• Operating System: Windows 11
• Browser and Version: not applicable

Log Output

This warning is logged from EngineExceptionHandlingMiddleware:

warn: Elsa.Workflows.Middleware.Workflows.EngineExceptionHandlingMiddleware[0]
      An exception was caught from a downstream middleware component
      System.ArgumentException: Must contain at least 1 request. (Parameter 'requests')
         at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
         at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
         at Elsa.MongoDb.Common.MongoDbStore`1.AddManyAsync(IEnumerable`1 documents, CancellationToken cancellationToken)
         at Elsa.Workflows.Runtime.Services.StoreWorkflowExecutionLogSink.PersistExecutionLogsAsync(WorkflowExecutionContext context, CancellationToken cancellationToken)
         at Elsa.Workflows.Runtime.Middleware.Workflows.PersistWorkflowExecutionLogMiddleware.InvokeAsync(WorkflowExecutionContext context)
         at Elsa.Workflows.Runtime.Middleware.Workflows.PersistActivityExecutionLogMiddleware.InvokeAsync(WorkflowExecutionContext context)
         at Elsa.Workflows.Runtime.Middleware.Workflows.PersistBookmarkMiddleware.InvokeAsync(WorkflowExecutionContext context)
         at Elsa.Workflows.Runtime.Middleware.Workflows.ScheduleBackgroundActivitiesMiddleware.InvokeAsync(WorkflowExecutionContext context)
         at Elsa.Workflows.Middleware.Workflows.EngineExceptionHandlingMiddleware.InvokeAsync(WorkflowExecutionContext context)

Troubleshooting Attempts

The root cause of the bug seems to be:

• DefaultAlterationRunner replaces the last middleware with the workflow execution pipeline with RunAlterationsMiddleware.
• RunAlterationsMiddleware runs the alterations on the workflow instance, but doesn't add any execution logs in collection WorkflowExecutionContext.ExecutionLog.
• Later in the workflow execution pipeline, PersistWorkflowExecutionLogMiddleware tries to persist the workflow execution logs from WorkflowExecutionContext.ExecutionLog, but WorkflowExecutionContext.ExecutionLog is empty. When using MongoDB for storage, MongoWorkflowExecutionLogStore.AddManyAsync(IEnumerable<WorkflowExecutionLogRecord> records, CancellationToken cancellationToken) throws an exception when it's passed an empty collection for parameter records.

The possible solutions would be to either:

• Add some execution logs for the alterations into WorkflowExecutionContext.ExecutionLog.
• In method StoreWorkflowExecutionLogSink.PersistExecutionLogsAsync(WorkflowExecutionContext context, CancellationToken cancellationToken), add a check that checks if WorkflowExecutionContext.ExecutionLog has any logs before attempting to persist them.

[rosca-sabina]

Hi @sfmskywalker! I would be happy to code a fix for this for 3.2.0, if you let me know which approach you prefer for the fix:

• Adding execution logs for alterations.
• Adding a check StoreWorkflowExecutionLogSink.PersistExecutionLogsAsync(WorkflowExecutionContext context, CancellationToken cancellationToken) to check if there are any logs to persist before attempting to persist them.
• A different approach.

[sfmskywalker]

Hi @rosca-sabina , thank you for reporting this issue and your offer to help. I think we should do both:

1. Add the necessary middleware to add execution logs, even for the alternation workflow (I'm not sure why this isn't happening, but would be great to get fixed)
2. At the lowest layer of the MongoDB persistence stack (MongobStore, it would be good to add a check first to see if there is anything to insert, as per your suggestion.

Thanks again!

[rosca-sabina]

Hello! I opened a PR with a fix for the issue, please let me know what you think: #5911.

I added a new property to class AlterationLogEntry, but if you think it's not necessary I can remove it.