conditional materialized view refresh and create-replace for the view

[kraftaa]

for issue: #88
hi I've started testing quary in the production and hitting dependencies errors: If I follow the logic of DROP/CREATE view I can't do it without CASCADE deleting all dependencies. The same with materialized views.

I suggest that for the view and materialized view there would be no DROP command and then changed creation commands:

• view : CREATE OR REPLACE VIEW ...
• materialized view: checking if the view exists first and based on that either REFRESH MATERIALIZED VIEW .. or CREATE MATERIALIZED VIEW ... In case of REFRESH there is no issue with dependent views.

thank you

[benfdking]

The fundamental problem with this approach is that if you change the model between the runs, the behavior is vastly different from what you would expect, e.g., changing the model as expected.

We do have a system for this, though, which I think we can rely on our "cached views" system. It's mainly defined in this file: https://github.com/quarylabs/quary/blob/main/rust/core/src/automatic_branching.rs.

We currently use it for our development flow; suppose you have models A, B, and C that depend on each other as follows, and you are pushing them to production in a materialized way.

A --> B --> C

It essentially measures the diff between your local setup and what's in production and tries to use as much "production" as possible to rely on materialized models as much as possible. So if your local models change C to C', you can still rely on production A and B because those are identical.

At its core, it's a "diffing system" that I think we can use here as well, to both speed up builds by

• running refresh if the model is materialized but the definition is the same
• skipping create/replace steps for views that are identical.

I hope some of the above makes sense.

[kraftaa]

Ben, thank you, yes it makes total sense.

[benfdking]

#141 The beginning!