[Bug] Invalid where filter for latest batch when event_column is of type date

[QMalcolm]

Currently, when running a microbatch model, the current time is right now. This, for the most part, just works. However, there is a problem.

Sometimes the event_time column is a date instead of a datetime. Now consider a model where the event_time is date_added which is a date type. The microbatch filter would then be something like date_added >= ‘2024-10-16 00:00:00’ and date_added < ‘2024-10-16 11:29:34’. Now, how does the data warehouse handle that? In some cases the datetime value gets auto truncated to a date, thus making the filter date_added >= ‘2024-10-16’ and date_added < ‘2024-10-16’. That is problematic because that filter will always return zero rows. To get around this, one solution could be to take the batch ceiling of the current time. That is, if our batch_size is day then 2024-10-16 11:29:34 would become 2024-10-17 00:00:00, and then upon truncation 2024-10-17.