Python dependency inference: add/emit debugging information

[lilatomic]

Is your feature request related to a problem? Please describe.
There currently isn't any information emitted about what dependencies were inferred. If dependency inference fails, developers don't have much to pinpoint the error.

For concrete usecases:

• I ran into an issue where dependent files weren't pulled into a test PEX. This was due to some shenanigans I had with pants source roots, where the imports were resolved from the repository root, not the closest parent source root.
• I ran into an issue where guarded imports weren't pulled into a test PEX. I had the incorrect module path, but the PEX built fine because they were weak imports

Describe the solution you'd like
A good first pass would be having the ability to output,

• for every file, the imports found and whether they were resolved.
• the map of 3rd party dependencies to modules
• the map of 1st party targets to modules

This would help developers confirm that the imports were found and understand if the targets of the import were found or ignored. For the example, with the second usecase, the imports found I would expect something like the following to let me know that it was deliberate and acceptable that there wasn't an error not resolving the imports:

[
  {
    "path": "//folder/file.py",
    "imports": [
      {
        "import": "a.b.c",
        "weak": true,
        "resolved": false
      }
    ]
  }
]

Describe alternatives you've considered
The current output is just checkpointing. With -ltrace, these are what is printed (many messages elided):

11:31:51.66 [DEBUG] Completed: Find all targets in the project
11:31:51.66 [DEBUG] Completed: Find all Python targets in project
11:31:51.66 [DEBUG] Completed: Creating map of third party targets to Python modules
11:31:51.67 [DEBUG] Completed: Creating map of first party Python targets to Python modules
11:31:51.67 [TRACE] Completed: Inferring Python dependencies by analyzing source
11:31:51.67 [TRACE] Completed: Inferring Python dependencies by analyzing source

Additional context
rel: #13283 ; this requests information on dependency inference, that requests graphing the normal dependencies. Discussion in that mentions graphing at different scopes, this mentions a scope below what could ordinarily be graphed by that.

[lilatomic]

I think there's a lot of information that will be output, so something like peek's --peek-output-file might be appropriate

[stuhood]

There is some (informal) prior art here in the JVM backend (and the go backend, as well, actually):

• https://github.com/pantsbuild/pants/blob/main/src/python/pants/jvm/goals/debug_goals.py
• https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/java/goals/debug_goals.py

Those goals dump the sources of thirdparty dependencies as JSON, and the exact extracted symbols per file (respectively).

If symbol extraction were standardized across languages behind a @union, a goal like this could be done in a language-agnostic manner. But failing that, adding a debug goal like this would also be an option. cc @tdyas

[Eric-Arellano]

I think that a debug_goals backend is a good idea for Python. This ticket would be awesome! Thanks for the suggestion.

[stuhood]

I think there's a lot of information that will be output, so something like peek's --peek-output-file might be appropriate

Oh, hm! It just occurred to me that another connection to your peek idea is #16967: essentially, you could think of the extracted imports / consumed-symbols of a file as effectively computed "file metadata" about that file. If we had additional generic computed per-file metadata like this, then peek might be a natural place to (optionally) render it...