[CT-2776] [Feature] Allow get_metadata_vars() to populate from env post-initial invocation

[gem7318]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

The requested feature from this change enables get_metadata_vars() to be populated if the environment is updated after the first time get_metadata_vars() is ever called.

Impetus is that in our deployments, we need to inject the project vars into the data.extra attribute of each Log Line to operate on for alerting down to the team / project / environment - mechanically, this means:

1. Get the post-parse project vars since they’re templated with pre-existing environment variables
2. Prepend them with DBT_ENV_CUSTOM_ENV_ and jam them into the runtime environment
3. Then call our build command so that the custom env vars make it into data.extra attribute of log lines it emits

Example

Post applying #7998, the following is possible:

# example
import os
from typing import Dict

from dbt.cli.main import dbtRunner, dbtRunnerResult

METADATA_ENV_PREFIX = "DBT_ENV_CUSTOM_ENV_"

def build() -> None:

    dbt: dbtRunner = dbtRunner()
    
    # Compile project to populate project `vars` with
    # template values from initial environment.
    dbt_run: dbtRunnerResult = dbt.invoke(["compile"])

    custom_vars: Dict[str, str] = {
        # Get the post-parse project vars from any results object.
        # Prefix them to be registered as `DBT_ENV_CUSTOM_ENV_*` vars.
        f"{METADATA_ENV_PREFIX}_{var_name}".upper(): var_value
        for result in dbt_run.result.results
        for var_name, var_value in result.node.config.meta.items()
        if (
            dbt_run.result and dbt_run.result.results
        )
    }
    
    # Update environment with project vars as environment vars.
    os.environ.update(custom_vars)

    # Build project with updated environment and custom env vars.
    # Without the change to `get_metadata_vars()`, the new vars
    # won't make it into the logger's context during this `build`.
    dbt.invoke(["build"])

More context

When using programmatic invocations, the dbt.core.events.functions.get_metadata_vars() function is called once when the dbtRunner is instantiated.

If no DBT_ENV_CUSTOM_ENV_* vars are found the first time it's called, it will continue to return an empty Dictionary upon subsequent invocations and will miss environment variables added after its initial instantiation.

The main consideration is if there are any ramifications from checking the environment each time get_metadata_vars() is called until it (possibly) becomes populated - this should only be a concern if get_metadata_vars() is called a zillion times and too costly to re-calculate on the fly.

Describe alternatives you've considered

Trying to get these project vars into the runtime environment before initial instantiation (untenable, we use project vars too heavily)

Who will this benefit?

Those using programmatic invocations and wanting to update the runtime environment across invocations of dbtRunner.invoke().

Are you interested in contributing this feature?

Here is a link to the required change that has been tested locally.

[gem7318]

Linking the PR associated with the change as a separate comment for clarity.

[jtcohen6]

@gem7318 Thanks for the super-thorough write-up!!

The change in #7998 seems straightforward enough. I'll queue it up for review by the eng team.

[gem7318]

@jtcohen6 Closing this issue as #7998 was merged🙏🏻