[CT-454] Add env var for ra3_node profile config

[jtcohen6]

Background

dbt-labs/dbt-core#3408 added support for cross-database (read-only) querying IFF the user is connecting to a cluster with RA3-type nodes. On the recommendation of the Redshift team, who do not anticipate "classic" node types disappearing anytime soon, and see them remaining the default for some time, we kept our preexisting error in place, and gave users a way to tell us (in profiles.yml) whether they're using RA3 or not. (Original discussion in dbt-labs/dbt-core#3179 + dbt-labs/dbt-core#3236)

Problem

dbt Cloud users do not have access to custom profiles.yml fields today, but they do have access to other configuration options (i.e. dbt_project.yml + env vars). Which of those is the right way forward?

1. "project" config (dbt_project.yml): I can't think of an exact precedent for this. Maybe quoting? Or the simple fact that sources have been defined in another warehouse? When negotiating the fuzzy boundary between configuration of the project and configuration of the warehouse, this feels more warehouse than project.
2. "global" config (flag + env var + profiles.yml): This feels more consistent with existing configurations.
   • profiles.yml already supports this config (though under a Redshift target, rather than the top-level config: block)
   • Env var is totally doable: DBT_REDSHIFT_RA3_NODE=True|False. If set via env var, this should take precedence over the profiles.yml setting, just like with global configs.
   • Flag would require a change to dbt-core (main.py). Eventually, we should aim to make this not the case, and allow plugins to register their own custom flags.

Tactical question: Which is the right spot to use the env var, if defined?

dbt-redshift/dbt/adapters/redshift/connections.py

Line 54 in 162dc30

ra3_node: Optional[bool] = False

dbt-redshift/dbt/adapters/redshift/impl.py

Line 72 in 162dc30

ra3_node = self.config.credentials.ra3_node

Much bigger question: Should we natively support env vars for all adapter-specific profiles.yml configs, via the formulation DBT_{ADAPTER}_{CONFIG}? If we could add this logic to dbt-core, in the Credentials base class, it would make it easier to support new configs as soon as they're added. I'm just not sure if this is the right-sized solution for the problem we have today in dbt Cloud, where it's difficult to coordinate versioned schemas of adapter-specific profiles.yml configuration. (I'll open a dbt-core issue for this)

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.