-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-1584] New top level commands: interactive compile #7008
Changes from all commits
94ac502
aa00da6
2f1a915
45d0b49
271b9b8
97db5d0
8c8d720
0430de7
b64662f
a5fcd10
7be301a
0df650e
9cb6a09
f1d0238
d684e53
0bc04ef
b02447b
dd0b361
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
kind: Features | ||
body: '[CT-1584] New top level commands: interactive compile' | ||
time: 2023-03-06T17:02:51.240582-08:00 | ||
custom: | ||
Author: aranke | ||
Issue: "6358" |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -30,6 +30,7 @@ | |
"FULL_REFRESH": False, | ||
"STRICT_MODE": False, | ||
"STORE_FAILURES": False, | ||
"INTROSPECT": True, | ||
} | ||
|
||
|
||
|
@@ -43,6 +44,7 @@ | |
"fail_fast", | ||
"indirect_selection", | ||
"store_failures", | ||
"introspect", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why we want to add introspect here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did it to help a test pass, but it might not be required. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Update: removing this doesn't help, so I'll leave it as-is for now. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you mean removing it would cause things to fail? This doesn't feel right given that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Punted into a new ticket: #7156 |
||
] | ||
|
||
|
||
|
Large diffs are not rendered by default.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1622,7 +1622,16 @@ message ConcurrencyLineMsg { | |
ConcurrencyLine data = 2; | ||
} | ||
|
||
// Skipped Q028 | ||
// Q028 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just curious, what is Q stand for? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Node execution, it's not the perfect category, but the best one I could find. |
||
message CompiledNode { | ||
string node_name = 1; | ||
string compiled = 2; | ||
} | ||
|
||
message CompiledNodeMsg { | ||
EventInfo info = 1; | ||
CompiledNode data = 2; | ||
} | ||
|
||
// Q029 | ||
message WritingInjectedSQLForNode { | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1711,7 +1711,13 @@ def message(self) -> str: | |
return f"Concurrency: {self.num_threads} threads (target='{self.target_name}')" | ||
|
||
|
||
# Skipped Q028 | ||
@dataclass | ||
class CompiledNode(InfoLevel, pt.CompiledNode): | ||
def code(self): | ||
return "Q028" | ||
|
||
def message(self) -> str: | ||
return f"Compiled node '{self.node_name}' is:\n{self.compiled}" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @dbeatty10 Does this message make sense, or should I change it to something else? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you have an example what this message would look like with realistic values? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yup:
|
||
|
||
|
||
@dataclass | ||
|
@@ -2306,7 +2312,7 @@ def message(self) -> str: | |
|
||
|
||
# The Note event provides a way to log messages which aren't likely to be useful as more structured events. | ||
# For conslole formatting text like empty lines and separator bars, use the Formatting event instead. | ||
# For console formatting text like empty lines and separator bars, use the Formatting event instead. | ||
@dataclass | ||
class Note(InfoLevel, pt.Note): | ||
def code(self): | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,6 +3,7 @@ | |
import time | ||
import traceback | ||
from abc import ABCMeta, abstractmethod | ||
from contextlib import nullcontext | ||
from typing import Type, Union, Dict, Any, Optional | ||
from datetime import datetime | ||
|
||
|
@@ -309,7 +310,7 @@ def skip_result(self, node, message): | |
|
||
def compile_and_execute(self, manifest, ctx): | ||
result = None | ||
with self.adapter.connection_for(self.node): | ||
with self.adapter.connection_for(self.node) if get_flags().INTROSPECT else nullcontext(): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If a user set There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, correct. |
||
ctx.node.update_event_status(node_status=RunningStatus.Compiling) | ||
fire_event( | ||
NodeCompiling( | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,7 @@ | |
) | ||
from dbt.events.test_types import IntegrationTestDebug | ||
|
||
|
||
# ============================================================================= | ||
# Test utilities | ||
# run_dbt | ||
|
@@ -160,6 +161,11 @@ def write_file(contents, *paths): | |
fp.write(contents) | ||
|
||
|
||
def file_exists(*paths): | ||
"""Check if file exists at path""" | ||
return os.path.exists(os.path.join(*paths)) | ||
|
||
|
||
Comment on lines
+164
to
+168
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not a problem to solve in this PR, but I wonder why There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We're already using There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yep, going with the flow make sense in this PR. Something for us to consider in a refactor down the road though. |
||
# Used in test utilities | ||
def read_file(*paths): | ||
contents = "" | ||
|
@@ -385,7 +391,6 @@ def check_relation_has_expected_schema(adapter, relation_name, expected_schema: | |
def check_relations_equal_with_relations( | ||
adapter: Adapter, relations: List, compare_snapshot_cols=False | ||
): | ||
|
||
with get_connection(adapter): | ||
basis, compares = relations[0], relations[1:] | ||
# Skip columns starting with "dbt_" because we don't want to | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we want to figure out a way to remove this section at some point. Just write down what I think, this works before we do that.