Bytewax materializer sets incremental materialization date before materialization completes

[james-crabtree-sp]

Expected Behavior

There should be an option to match the LocalMaterializer behavior, which waits for materialization to finish before updating the incremental materialization timestamp for a feature view

Current Behavior

The incremental materialization timestamp is updated as soon as the materialization job is successfully created, even though that job may not successfully complete. This means that some updates will never be materialized to the online store.

Steps to reproduce

Set the batch_engine image to an image that immediately exits with exit code 1. Run an incremental materialization with end date set to current time. Observe that the pods and job fail. Run an incremental materialization again. Observe that the start date is now the end date of the last run.

Specifications

• Version: 0.31
• Platform: fedora linux
• Subsystem: bytewax batch_engine

Possible Solution

Optionally allow for a synchronous materialization in bytewax, which will wait for a feature view materialization to finish before returning success and moving on to the next feature view and throwing an exception if the job fails. This will match the behavior of the LocalMaterializer