Manifest API

[cmcarthur]

We need to make some key changes to the Manifest for future compatibility with the dbt api.

• class: Manifest
  • one key change here is that there’s no longer one manifest object for each status (UnparsedManifest, ParsedManifest, CompiledManifest, etc.). since the manifest can contain nodes with heterogeneous statuses, there is only one manifest, and we check the parsing / compilation status of a node instead of the entire manifest.
  • loading & parsing
    • this set of functions provides utilities for loading source code into dbt’s core API, and parsing that code into a usable manifest.
    • get_all_metadata()
      • returns a dictionary of path: metadata for all source code loaded into dbt
    • get_metadata(path)
      • returns the metadata for the source code at path. useful for checking whether or not a file needs to be updated.
    • delete(path)
      • mark the source code at path deleted
    • incorporate_source_code(path, contents, metadata, source_code_type)
      • incorporates a source file change into the manifest. returns the updated manifest.
      • path: a unique identifier for this snippet of dbt source code. could be a filesystem path, or some other unique identifier in non-filesystem-based environments.
      • contents: the contents of this dbt source code snippet.
      • metadata: arbitrary metadata to associate with this node. this can be implemented by the client. for the filesystem client, it’s suggested to include a file hash and modified at timestamp.
      • source_type: the type of source code being passed in, e.g. ‘schema_yml’, ‘dbt_project_yml’, ‘model_sql’, ‘test_sql’, etc.
    • generate_all_nodes()
      • processes all of the changed source code entries in the manifest into nodes, and creates all of the edges between those nodes so that they can be parsed. returns the updated manifest.
    • parse_all_nodes()
      • processes all of the changed nodes in the manifest. this should not be executed until the entire project is loaded! if all nodes are already parsed, this is a no-op. returns the updated manifest.
    • private _generate_node_from_path(path)
      • given a source code path, generate the nodes for that path. if the path is deleted, this will remove the nodes created from that path. returns the list of generated nodes
    • private _parse_node(unique_id)
      • parses a single node by:
        • if this node is parsed, move on
        • recursively parsing all of the dependencies of this node
        • parsing this node

[drewbanin]

closing as not relevant anymore