Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add batching to sources and fix batch size for tests #348

Merged

Conversation

samw430
Copy link
Contributor

@samw430 samw430 commented May 26, 2023

Overview

  • Adds batching to sources upload
  • Decreases batch size for tests upload

Update type - breaking / non-breaking

  • Minor bug fix
  • Documentation improvements
  • Quality of Life improvements
  • New features (non-breaking change)
  • New features (breaking change)
  • Other (non-breaking change)
  • Other (breaking change)

What does this solve?

15:53:26 15:53:26 Running 1 on-run-end hook 15:53:26 Uploading model executions 15:53:28 Uploading seed executions 15:53:28 Uploading snapshot executions 15:53:28 Uploading test executions 15:53:28 Uploading exposures 15:53:31 Uploading tests 15:53:46 BigQuery adapter: https://console.cloud.google.com/bigquery?project=cue-dlake-us-develop&j=bq:US:f717fe57-be4c-4184-9a34-bf3775d17d91&page=queryresults 15:53:46 Database error while running on-run-end 15:53:46 15:53:46 Finished running 11 incremental models, 23 view models in 0 hours 0 minutes and 39.34 seconds (39.34s). 15:53:46 15:53:46 Completed with 1 error and 0 warnings: 15:53:46 15:53:46 on-run-end failed, error: 15:53:46 The query is too large. The maximum standard SQL query length is 1024.00K characters, including comments and white space characters.

Outstanding questions

Anything else I should test beyond running the integration tests against BQ?

What databases have you tested with?

  • Snowflake
  • Google BigQuery
  • Databricks
  • Spark
  • N/A

@samw430 samw430 had a problem deploying to Approve Integration Tests May 26, 2023 17:21 — with GitHub Actions Failure
@samw430 samw430 had a problem deploying to Approve Integration Tests May 26, 2023 17:21 — with GitHub Actions Failure
@samw430 samw430 had a problem deploying to Approve Integration Tests May 26, 2023 17:21 — with GitHub Actions Failure
macros/upload_results.sql Outdated Show resolved Hide resolved
macros/upload_results.sql Outdated Show resolved Hide resolved
@samw430 samw430 temporarily deployed to Approve Integration Tests May 30, 2023 17:42 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 30, 2023 17:42 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 30, 2023 17:42 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 14:49 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 14:49 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 14:49 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 14:49 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 14:49 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 14:49 — with GitHub Actions Inactive
macros/upload_results.sql Outdated Show resolved Hide resolved
Co-authored-by: Jared Rimmer <100997264+jared-rimmer@users.noreply.github.com>
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 16:15 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 16:15 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests May 31, 2023 16:15 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests June 1, 2023 12:35 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests June 1, 2023 12:35 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests June 1, 2023 12:35 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests June 1, 2023 12:35 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests June 1, 2023 12:35 — with GitHub Actions Inactive
@samw430 samw430 temporarily deployed to Approve Integration Tests June 1, 2023 12:35 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer merged commit f0617cf into brooklyn-data:main Jun 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants