Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix missing value in upload model executions - add else clause to for if loops #386

Conversation

kenpokiwi
Copy link
Contributor

@kenpokiwi kenpokiwi commented Sep 21, 2023

Overview

See details in #385

Update type - breaking / non-breaking

  • Minor bug fix
  • Documentation improvements
  • Quality of Life improvements
  • New features (non-breaking change)
  • New features (breaking change)
  • Other (non-breaking change)
  • Other (breaking change)

What does this solve?

Resolves #385

Outstanding questions

What databases have you tested with?

  • Snowflake
  • Google BigQuery
  • Databricks
  • Spark
  • Postgres
  • N/A

@kenpokiwi kenpokiwi temporarily deployed to Approve Integration Tests September 21, 2023 01:53 — with GitHub Actions Inactive
@kenpokiwi kenpokiwi had a problem deploying to Approve Integration Tests September 21, 2023 01:53 — with GitHub Actions Failure
@kenpokiwi kenpokiwi temporarily deployed to Approve Integration Tests September 21, 2023 01:53 — with GitHub Actions Inactive
@kenpokiwi kenpokiwi changed the title add else clause to for if loops Fix missing value in upload model executions - add else clause to for if loops Sep 21, 2023
@@ -179,6 +179,8 @@
{% else %}
'{{ stage.started_at }}', {# compile_started_at #}
{% endif %}
{ % else %}
null, {# compile_started_at #}
{% endfor %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think originally this loop.length section was designed for the use-case you are fixing here, but doesn't make sense (it wouldn't run if the loop was empty). That means it can be removed I believe. Looking at the logic here, it also looks like it could be hugely simplified using Jinja filtering:

                {% set compile_started_at = (model.timing | selectattr("name", "eq", "compile") | first | default({})).get("started_at") %}
                {% if compile_started_at %}'{{ compile_started_at }}'{% else %}null{% endif %}, {# compile_started_at #}
                {% set query_completed_at = (model.timing | selectattr("name", "eq", "execute") | first | default({})).get("completed_at") %}
                {% if query_completed_at %}'{{ query_completed_at }}'{% else %}null{% endif %}, {# query_completed_at #}

I think this would aid readability and consistency. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think that is much more elegant :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking at this. For consistency, please could you also add the same logic into the other two macros in here for BQ and Snowflalke? From lines 88 and 164. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. Update done.

Refactor logic to evaluate compile_started_at and query_completed_at values.
@kenpokiwi kenpokiwi had a problem deploying to Approve Integration Tests September 21, 2023 21:44 — with GitHub Actions Failure
@kenpokiwi kenpokiwi had a problem deploying to Approve Integration Tests September 21, 2023 21:44 — with GitHub Actions Failure
@kenpokiwi kenpokiwi temporarily deployed to Approve Integration Tests September 21, 2023 21:44 — with GitHub Actions Inactive
@kenpokiwi kenpokiwi had a problem deploying to Approve Integration Tests September 26, 2023 19:02 — with GitHub Actions Failure
@kenpokiwi kenpokiwi temporarily deployed to Approve Integration Tests September 26, 2023 19:02 — with GitHub Actions Inactive
@kenpokiwi kenpokiwi had a problem deploying to Approve Integration Tests September 26, 2023 19:02 — with GitHub Actions Failure
@glsdown glsdown had a problem deploying to Approve Integration Tests September 28, 2023 10:21 — with GitHub Actions Failure
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 10:21 — with GitHub Actions Inactive
@glsdown glsdown had a problem deploying to Approve Integration Tests September 28, 2023 10:21 — with GitHub Actions Failure
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:20 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:20 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:20 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown glsdown temporarily deployed to Approve Integration Tests September 28, 2023 13:24 — with GitHub Actions Inactive
@glsdown
Copy link
Contributor

glsdown commented Sep 28, 2023

Thanks for all your work on this!

@glsdown glsdown merged commit 3cf1473 into brooklyn-data:main Sep 28, 2023
11 checks passed
@glsdown glsdown mentioned this pull request Sep 29, 2023
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Error from snowflake__get_model_executions_dml_sql missing value
2 participants