Maintain global provider schema cache

[radeksimko]

Past/Current Situation

Currently (v0.11.0) we obtain provider schemas via the CLI (terraform providers schema -json) and track it for each module individually. This is suboptimal especially in cases of more root modules with duplicate provider versions as we store the same data more than once. More importantly we have no way of cross-referencing and reusing the data.

LS assumes that there's a single source of truth for the schema, when sources could in reality be combined.

Global tracking was initially avoided, because of the limited ways of identifying a provider (there was no concept of provider "address" until Terraform 0.13) and because the first architecture of LS didn't account for the complexity of module nesting nor it parsed provider requirements nor had capability to detect installed provider versions.

We have gathered feedback from GitHub and VS Code Marketplace in the past months where users were confused by various warnings we issue which can be attributed in one way or another to lack or inaccuracy of provider schema and practically very limited functionality of the language server.

• Alternative root modules found for <filename> (project), picked: <something>. You can try setting paths to root modules explicitly in settings. vscode-terraform#524
• Repeated message "No root module found for <filename> functionality may be limited" #221
• Still getting "No root module found for ..." vscode-terraform#447
• Workspace not initialized vscode-terraform#329

We started pre-baking some schemas to address this, which provided at least some data to these users, but the solution is generally very naive (which is not a criticism, but a chosen trade-off) and it's becoming clear that we'll need a more flexible solution, especially once we start pulling schemas from more sources per #193 and #355

Proposal

We now (since 0.13 was released) have a mechanism for identifying providers, whether they are installed locally or come from a Registry (via an address).

It seems natural to therefore allow combining provider schemas, which we do to a limited degree already by falling back to preloaded schemas.

Here is what will likely need to be done:

1. create a global cache which can hold unique schema in a queryable cache
   • provider identity (e.g. registry.terraform.io/hashicorp/azurerm)
   • version (e.g. 2.40.0)
   • source ? (e.g. path to the module where terraform providers schema -json ran, or preloaded JSON export)
2. parse provider requirements for any loaded module, so we know which versions to pick
3. create a mechanism for picking versions based on:
   • parsed requirements
   • source, e.g.
     • locally sourced schema is preferred
     • when two locally sourced schemas of different version are available, hierarchy proximity is considered, as also explained in Use module version proximity in finding relevant schema #180
     • higher versions are generally preferred
4. remove Alternative root modules found warning and log this internally instead
5. remove No root modules found warning and log this internally instead

[satish-chef]

Does this fix the "No schema found for "abc/xyz/def". Functionality may be limited. You may need to run terraform init." in Vscode. If yes, please describe what changes to do in settings.json.

[radeksimko]

@satish-chef This patch has not been shipped yet - we plan to ship it this week or beginning of the next one and VS Code users will receive the update automatically.

Thank you for your patience.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the context necessary to investigate further.