Name resolution

[Y-Nak]

Name Resolution and Import Resolution

This PR implements import resolution and early path resolution. Accompanying these main features, this PR also introduces the Visitor and ScopeGraph at the HIR level and the AnalysisPass trait for guaranteeing the orthogonality of diagnostics as a public API. As this is a substantially large PR, I'll break down and explain the overview of the language specification about name resolution and public API and provide a summarized description of the implementation details for each item.

Name Resolution Specification

General Concept

1. Scope

Name resolution is determined by the relation between the scope where a name is defined and the scope referencing the name.
A new scope is created for each item (e.g., fn or mod) or Block Expression ({}) ¹.
If the scopes differ, definitions with the same names are treated as independent and will not collide.
Example:

// Scope0
// ...
mod foo {
    // Scope1
    pub fn foo() {
        // Scope2
        {
            // Scope3
        }
    }
}

2. NameDomain

In Fe, the definitions that are targeted for name resolution are classified into the following three domains:

• Type Domain: Types/Traits, Modules, Generic Params
• Value Domain: Functions, Constants, Enum Variants
• Field Domain: Struct Fields
  Definitions belonging to different domains can have the same name even within the same scope without collision.
  Example:

// Value Domain
pub fn Foo() {
    ...
}

/// Type Domain
pub struct Foo() {
    ...
}

Also, definitions belonging to different domains can be imported into the same scope.
Example:

pub enum MyEnum {
    Foo
    Bar
}

pub struct Foo {}

mod mod1 {
    use super::MyEnum::Foo // Value Domain
    use super::Foo // Type Domain
}

3. Shadowing

The following ordered elements affect shadowing in name references:

1. Definitions in the same scope as the current
2. Definitions imported by named import into the same scope as the current scope
3. Definitions imported by glob import into the same scope as the current scope
4. Definitions obtained by applying the same shadowing rules in the lexical parent scope of the current scope
5. Names of external ingots
6. Compiler-builtin definitions (e.g., u32 or i64)

Smaller numbers have higher priority. i.e., 1 > 2 > 3 > 4 > 5 > 6. e.g., 1 shadows 2, 3, 4, 5, and 6.

Example:

mod mod1 {
    pub struct Foo {}
}

mod mod2 {
    pub struct Foo {}
}

// Order 1.
struct Foo {}

/// Order 2.
use mod1::Foo

/// Order 3.
use mod2::*

struct MyS {
    /// Foo(Order 1.) is used.
    x: Foo
}

NOTE: Modules work as boundaries for lexical scopes. Therefore, to reference outside of a module, you need to use super or an absolute path.
Example:

struct Foo {}
mod mod1 {
    use super::Foo // Ok
    use Foo // Error
}

4. Visibility

Currently, Visibility exists only as public and private². To specify visibility as public, pub keyword can be used at the definition site.
If there is no pub keyword, the item is treated as private.
Example:

pub fn foo() {} // public
fn bar() {} // private

Public definitions can be referenced from all scopes. Private definitions can only be referenced from the scope where they are defined and their child scopes.
Example:

fn foo() {
    mod1::foo() // Ok
    mod2::priv_foo() // Error
}

mod mod1 {
    pub fn foo() {}
    fn priv_foo() {}
}

As an exception, fields, even if they are private, can be referenced from the scope that contains there parent struct.
Example:

fn foo() {
    let x = S { x: 0 } // Ok
}

struct S {
    x: i32 // Private field defined in the scope of S.
}

NOTE: Definition visibility introduced into a scope with use is overwritten by the visibility of use.
Example:

fn foo() {
    let x = mod1::bar() // Error.
}

mod mod1 {
    use mod2::bar // Overwrite visibility as private.

    mod mod2 {
        pub fn bar() {} // Defined as public.
    }
}

Import Resolution Specification

1. Named Import

A Named Import is a standard import that imports a single definition from a certain scope into the scope.
Example:

use mod1::foo

mod mod1 {
    pub fn foo() {}
}

2. Glob Import

A Glob Import introduces all visible definitions from a certain scope into the scope.

Example:

use mod1::* // `mod1::inner1` and `mod2::inner2` are imported.

fn foo {
    inner1() // Ok
    inner2() // Ok
    inner3() // Error
}

mod mod1 {
    pub fn inner1() {}
    pub fn inner2() {}
    fn inner3() {}
}

3. Importability

Definitions belonging to the Type Domain and Value Domain can be imported using use.
NOTE: Enum Variant belongs to the Value Domain, hence can be imported using use.
Example:

use mod1::Foo // Ok
use Bar::{A, B} // Ok
use mod1::Foo::x // Error

mod mod1 {
    pub struct Foo {
        x: i32
    }

    pub enum Bar {
        A
        B
    }
}

However, types that are defined additively, such as associated types, can not be imported using use.
Example:

use Foo::Bar // Error. `Bar` is an associated type of `Foo`.

struct Foo { }

trait Trait {
    type Bar
}

impl Trait for Bar {
    type Bar = Foo
}

4. Use Aliases

Use Alias allows users to define an alias name for an imported definition.
The as keyword is used for this purpose.
NOTE: aliases can only be defined for Named Imports.
Example:

use mod1::Bar

mod mod1 {
    pub use mod2::Foo as Bar
    pub use mod2::* as Bar // Error

    mod mod2 {
        pub struct Foo {}
    }
}

5. Non-Cyclic Dependency between Imports

A definition imported by use can be used by another use in the same scope.
Example:

use mod1::*
use mod2::Foo // `mod2` is imported by `use mod1::*`.

mod mod1 {
    pub mod mod2 {
        pub struct Foo {}
    }
}

6. Circular Imports

In Fe, modules can reference each other.
Example:

pub mod mod1 {
    use super::mod2::Bar
    pub struct Foo {}
}

pub mod mod2 {
    use super::mod1::Foo
    pub struct Bar {}
}

Circular imports using glob are also possible.
Example:

pub mod mod1 {
    // `Foo`, `Bar`, and `BarImported` are visible in this scope.
    pub use super::mod2::Bar as BarImported
    pub use super::mod2::*

    pub struct Foo {}
    
}

pub mod mod2 {
    // `Foo`, `Bar`, `BarImported`, and `BarPriv` are visible in this scope.
    pub use super::mod1::*

    pub struct Bar {}
    
    struct BarPriv {}
}

However, if a cycle occurs between Named Imports, an error occurs.
Example:

pub mod mod1 {
    pub use super::mod2::Foo as Bar // Error
}

pub mod mod2 {
    pub use super::mod1::Bar as Foo // Error
}

7. Import Specific Ambiguity

The algorithm used in import resolution could introduce ambiguity errors specific to use.
This specific ambiguity error occurs when the following two conditions are met:

• A name imported via Glob Import is referenced by a Named Import, and
• That name conflicts with a definition that is implicitly introduced into the scope (i.e., an external ingot name or a compiler built-in definition)

Example:

use mod1::*
use i32::Foo // Error. i32 is also a built-in type; hence `i32` is regarded as ambiguous.

mod mod1 {
    pub mod i32 {
        pub struct Foo {}
    }
}

Public API for LSP

PathResolution

1. ScopeId

[fe_hir::hir::scope_graph::ScopeId] is a reference to [fe_hir::hir::scope_graph::Scope], which is uniquely defined for a scope. In name resolution, the information about the reference scope is always necessary, thus ScopeId is always passed to the name resolution solver. Since all Hir items can be embedded into ScopeId, it is possible to obtain ScopeId from Hir::ItemKind via the ScopeId::from_item method.

2. NameRes and NameResBucket

hir_analysis::name_resolution::NameRes and hir_analysis::name_resolution::NameResBucket are types used to represent the result of name resolution. The definitions of NameRes and its related types are as follows.

/// The struct contains the lookup result of a name query.
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct NameRes {
    /// The kind of the resolution.
    pub kind: NameResKind,
    /// The domain of the name resolution.
    pub domain: NameDomain,
    /// Where the resolution is derived from. (e.g, via `use` or item definition
    /// in the same scope).
    pub derivation: NameDerivation,
}

/// Each resolved name is associated with a domain that indicates which domain
/// the name belongs to.
/// The multiple same names can be introduced in the same scope as long as they
/// are in different domains.
///
/// E.g., A `Foo` in the below example can be introduced in the same scope as a
/// type and variant at the same time.
/// ```fe
/// struct Foo {}
/// enum MyEnum {
///     Foo
/// }
/// use MyEnum::Foo
/// ```
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum NameDomain {
    /// The domain is associated with all items except for items that belong to
    /// the `Value` domain.
    Type = 0b1,
    /// The domain is associated with a local variable and items that are
    /// guaranteed not to have associated names. e.g., `fn`, `const` or enum
    /// variables.
    Value = 0b10,
    /// The domain is associated with struct fields.
    Field = 0b100,
}

/// The name resolution kind obtained from a name resolver.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, derive_more::From)]
pub enum NameResKind {
    /// The name is resolved to a scope.
    Scope(ScopeId),
    /// The name is resolved to a primitive type.
    Prim(PrimTy),
}

/// The name derivation indicates where a name resolution comes from.
/// Name derivation is used to track the origin of a resolution, and to
/// determine the shadowing rules.
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub enum NameDerivation {
    /// Derived from a definition in the current scope.
    Def,
    /// Derived from a named import in the current scope.
    NamedImported(Use),
    /// Derived from a glob import in the current scope.
    GlobImported(Use),
    /// Derived from a lexical parent scope.
    Lex(Box<NameDerivation>),
    /// Derived from an external ingot.
    External,
    /// Derived from a builtin primitive.
    Prim,
}

NameResBucket is a type that collects NameRes for each domain. The result of name resolution is held for each domain, so NameResolutionResult<NameRes> is defined for each domain. NameResBucket has several utility methods defined. For more details, please refer to their doc comments.

/// The struct contains the lookup result of a name query.
/// The results can contain more than one name resolutions which belong to
/// different name domains.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct NameResBucket {
    bucket: FxHashMap<NameDomain, NameResolutionResult<NameRes>>,
}

3. Path Resolution

Currently, the defined entry points for name resolution/path resolution are the following functions defined in hir_analysis::name_resolution::mod.rs.

/// Resolves the given path in the given scope.
/// It's not necessary to report any error even if the `EarlyResolvedPath`
/// contains some errors; it's always reported from [`PathAnalysisPass`].
pub fn resolve_path_early(
    db: &dyn HirAnalysisDb,
    path: PathId,
    scope: ScopeId,
) -> EarlyResolvedPath {
    ...
}

/// Resolves the given path segments in the given scope.
/// It's not necessary to report any error even if the `EarlyResolvedPath`
/// contains some errors; it's always reported from [`PathAnalysisPass`].
pub fn resolve_segments_early(
    db: &dyn HirAnalysisDb,
    segments: &[Partial<IdentId>],
    scope: ScopeId,
) -> EarlyResolvedPath {
    ...
}

The early postfix indicates that this is a name resolution at an early stage before resolving type associated items. Name resolution related to types requires context for trait resolution and type inference, but the execution of trait resolution and type inference requires early name resolution. Therefore, the return type of these functions, EarlyResolvedPath, is defined as a union type as follows.

/// The result of early path resolution.
/// There are two kinds of early resolution results:
/// 1. Fully resolved path, which is a path that is fully resolved to concrete
///    items.
/// 2. Partially resolved path. This happens when the path is partially resolved
///    to a type, and the rest of the path depends on the type to resolve.
///    Type/Trait context is needed to resolve the rest of the path.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum EarlyResolvedPath {
    Full(NameResBucket),

    /// The path is partially resolved; this means that the `resolved` is a type
    /// and the following segments depend on type to resolve.
    /// These unresolved parts are resolved in the later type inference and
    /// trait solving phases.
    Partial {
        res: NameRes,
        unresolved_from: usize,
    },
}

AnalysisPass

hir::analysis_pass::AnalysisPass is a trait for marking Analysis that could potentially generate Diagnostics, while maintaining the orthogonality of Diagnostics.
This can also be considered a thin wrapper to address the problem arising from the fact that the semantic boundaries of each Diagnostics and the boundaries of salsa-tracked functions do not necessarily coincide.
For example, hir_analysis::PathAnalysisPass and hir_analysis::DefConflictAnalysisPass are different AnalysisPasses, but their internal implementations invoke the same salsa-tracked function.
These AnalysisPasses can be managed as follows by hir::analysis_pass::AnalysisPassManager.

let mut pass_manager = AnalysisPassManager::new();
// Add analysis passes to the pass manager.
pass_manager.add_module_pass(Box::new(ParsingPass::new(db)));
pass_manager.add_module_pass(Box::new(DefConflictAnalysisPass::new(db)));
pass_manager.add_module_pass(Box::new(ImportAnalysisPass::new(db)));
pass_manager.add_module_pass(Box::new(PathAnalysisPass::new(db)));

// Obtain the result of registered analysis passes on the given module.
let diag = pass_manager.run_on_module(db, module_id);

Visitor

hir::visitor is an implementation of the Visitor Pattern for visiting HIR nodes.
For more about Visitor Pattern, please refer to Rust Design Pattern or here.

1. VisitorCtxt

One difference between the common Visitor implementation and the Visitor implemented in HIR is that all visit_* methods and walk_* functions take VisitorCtxt as an argument.
VisitorCtxt manages the LazySpan and ScopeId of the nodes being visited. As VisitorCtxt is automatically updated within walk_*, users do not need to explicitly update VisitorCtxt.
However, users need to inject VisitorCtxt from the outside to initiate the Visitor's traversal. VisitorCtxt can be constructed from all Item or Expr/Stmt/Pat nodes.

Example:

use fe_hir::visitor::prelude::*;

/// Collect all paths and their corresponding span.
#[derive(Default)]
struct PathCollector {
    paths: Vec<(PathId, LazyPathSpan)>,
}

impl Visitor for PathCollector {
    fn visit_path(&mut self, ctxt: &mut VisitorCtxt<'_, LazyPathSpan>, path: PathId) {
        // Collect path and its span.
        self.paths.push((path, ctxt.span().unwrap()));
    }
}

/// Collect all paths in the given `top_mod`(`top_mod` is a root module of a
/// file).
fn collect_paths_in_module(db: &dyn HirDb, top_mod: TopLevelMod) -> Vec<(PathId, LazyPathSpan)> {
    // Construct VisitorCtxt from the given top_mod.
    let mut ctxt = VisitorCtxt::with_top_mod(db, top_mod);
    let mut visitor = PathCollector::default();
    visitor.visit_top_mod(&mut ctxt, top_mod);
    visitor.paths
}

Implementation Details

ScopeGraph

hir::hir_def::scope_graph::ScopeGraph is built for each TopLevelModule, which corresponds to (.fe file), representing relationships between HIR nodes.
Please refer to A Theory of Name Resolution for an overview of ScopeGraph.
NOTE: The name resolution algorithm introduced by this paper is different from the name resolution algorithm in Fe. Please refer only to the structure of the scope graph.

In name resolution, besides relationships between lexical scopes, dependencies between modules introduced by keywords such as super need to be considered. Therefore, the following types of edges are defined in ScopeGraph.

/// A specific edge property definition.
///
/// NOTE: The internal types of each variant contain very small amounts of
/// information, the reason why we need to prepare each internal type is to
/// allow us to implement traits to each edge directly.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, derive_more::From)]
pub enum EdgeKind {
    /// An edge to a lexical parent scope.
    Lex(LexEdge),
    /// An edge to a module.
    Mod(ModEdge),
    /// An edge to a type.
    Type(TypeEdge),
    /// An edge to a trait.
    Trait(TraitEdge),
    /// An edge from a scope to a generic parameter.
    GenericParam(GenericParamEdge),
    /// An edge to a value. The value is either a function or a
    /// constant.
    Value(ValueEdge),
    /// An edge to a field definition scope.
    Field(FieldEdge),
    /// An edge to an enum variant definition scope.
    Variant(VariantEdge),
    /// An edge to a module that is referenced by a `super` keyword.
    Super(SuperEdge),
    /// An edge to an ingot that is referenced by an `ingot` keyword.
    Ingot(IngotEdge),
    /// An edge to a scope that is referenced by a `self` keyword.
    Self_(SelfEdge),
    /// An edge to a scope that is referenced by a `Self` keyword.
    SelfTy(SelfTyEdge),
    /// An edge to an anonymous scope, e.g., `impl` or function body.
    Anon(AnonEdge),
}

EdgeKind is used by the NameResolver and is also used when traversing HIR nodes.
NOTE: By decoupling the ScopeGraph from the definition of HIR nodes, the definition of HIR nodes themselves are scope-agnostic.

It's possible to visualize ScopeGraph with the following command.
cargo run --bin fe-driver2 [fe-file.fe] --dump-scope-graph

Here is an example of a ScopeGraph for the following Fe file.

// lib.fe
pub struct S0 {}

mod inner {
    struct S1<T> {
        t: T
    }
    
    mod inner_most {
        struct S2 {}
    }
    
    pub fn func<T>(label1 x: i32, label2: T) {
        use super::S0
        struct S3 {
            s0: S0
        }
    }
    
}

Name Resolver

hir_analysis::name_resolution::name_resolver::NameResolver is the core implementation for name resolution.
The entry point for NameResolver is NameResolver::resolve_query(&mut self, query: NameQuery) -> NameResBucket.
NameQuery is defined as follows.

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct NameQuery {
    /// The name to be resolved.
    name: IdentId,
    /// The scope where the query is emitted.
    scope: ScopeId,
    directive: QueryDirective,
}

QueryDirective is a type to specify the policy for the search in the name resolution engine and is defined as follows.

/// The query directive is used to control the name resolution behavior, such as
/// whether to lookup the name in the lexical scope or not.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct QueryDirective {
    /// If `allow_lex` is `true`, then the query will be propagated to the
    /// lexical scope if the name is not found in the current scope.
    allow_lex: bool,

    /// If `allow_external` is `true`, then the query will be propagated to the
    /// external ingot and builtin types as well.
    allow_external: bool,

    /// If `allow_glob` is `true`, then the resolver uses the glob import to
    /// resolve the name.
    allow_glob: bool,
}

NOTE: Note that NameResolver itself does not have the concept of Path as can be seen from the implementation of resolve_query.

The behavior of NameResolver is very simple.
It identifies the destination Scope for the query by asking QueryPropagator if the query should be propagated to the next scope or not.
QueryPropagator is a trait that is implemented for each EdgeKind type.

The definition of QueryPropagator is as follows.

/// The propagator controls how the name query is propagated to the next scope.
trait QueryPropagator {
    fn propagate(self, query: &NameQuery) -> PropagationResult;
    fn propagate_glob(self) -> PropagationResult;
}

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum PropagationResult {
    /// The query is resolved to the next scope(edge's destination).
    Terminated,
    /// The query resolution should be continued, i.e., the query is propagated
    /// to the next scope and the next scope should be searched for the query.
    Continuation,
    /// The query can't be propagated to the next scope.
    UnPropagated,
}

NOTE: Imported resolutions are not reflected in the ScopeGraph. Therefore, NameResolver also need to use Importer to reflect them in name resolution.
For more information about the Importer, please refer to the section on Import Resolution

Import Resolution

In Fe v2, since use is allowed to be interdependent, ImportResolver uses fixed-point computation to repeatedly perform name resolution until the set of Imported resolutions converges. Specifically, it applies import resolution repeatedly until ScopeState, defined in each scope, changes to the Closed state in all scopes, or there are no changes in the iteration of import resolution. The Import resolutions in each iteration are added to the IntermediateResolvedImport set. This set is passed to the NameResolver, allowing the use path segment to be resolved with the latest imported set. IntermediateResolvedImport is abstracted by implementing the Importer trait, so NameResolver is independent of whether the Query to be resolved is in the context of import resolution.

NOTE: To ensure reaching a fixed point, IntermediateResolvedImport does not allow operations other than the insertion to the set.

Proc macro for keywords prefilling

The fe-macro crate is the implementation of the procedural macros used in fe-v2. Currently, only the define_keywords macro for prefilling keyword identifiers into the DB is implemented. Identifiers pre-filled by the define_keywords macro can be used as constants. Please refer to hir::hir_def::IdentId.

In the future, test-related macros are also planned to be implemented in this crate.

uitest

The uitest crate is a crate for conducting integration tests related to the Diagnostics of fe-v2. All test cases included here cause compiler errors and are intended to verify whether errors are correctly reported to users. Currently, we are using insta to manage snapshots of test cases, but in the future, we want to provide functionality equivalent to uitest as implemented in rustc.

Footnotes

1. Implementation-wise, scopes are defined at a more granular level (for instance, Generic Parameters have their own scope), but this does not affect the language specification. ↩

2. In the future, other visibility options such as crate or super may be introduced. ↩

[sbillig]

@Y-Nak all three cycle examples result in an infinite loop when used as tests (the code blocks in 6. Circular Imports)

[Y-Nak]

Hmm, I broke something in resolving conflicts or the refactoring process. I'll check it and add tests for the cases.

[Y-Nak]

@sbillig I fixed it and added tests for them in Fix a bug that import resolver falls into infinite loop.
Also, in Improve readability of snap files for import test, I improved readability of snap files in import test, which is nothing to do with the bug.

[Y-Nak]

BTW, the first example of the circular imports section describes an error, but I meant to write valid circular imports.
I rewrite it to a valid case; you can find the original example here.

[Y-Nak]

@sbillig Oops, the example in 7. Import Specific Ambiguity was not appropriate, i32 inside mod1 was specified as private, so the resolver doesn't cause an ambiguity error(instead, the resolver raises an error becausei32 type can't be used in the middle of use path, only enum and mod is allowed to be used in the middle of use path).
I rewrote it by adding pub to the i32 in the mod1.

[Y-Nak]

Remove the use from the suspicious import set when it turns out to cause another error other than the import-specific ambiguous error that is described in 7. Import Specific Ambiguity.

[Y-Nak]

^ The main change in Remove duplicated error messages by cleaning suspicious import set.

[Y-Nak]

Oh, I realized that we need to add some edge kinds, e.g., to represent methods defined in impl block, since we need to reject the code below when we introduce a function type.

struct Foo {};
impl Foo {
     pub fn foo() {}
     pub fn bar(f: foo) {}
}

But I guess it'd be ok to postpone it since it doesn't affect the name resolution algorithm itself.

[Y-Nak]

I refactored the verify_ambiguity method in Refactor import ambiguity verification function.
And this is out of the scope of this PR, but I added a minor improvement for parsing recovery in Improve parsing recovery policy in function parsing.

@micahscopes Please use the Improve parsing recovery policy in function parsing, sorry for the inconvenience.

[sbillig]

This is fucking dope

[Y-Nak]

@sbillig Thanks for reviewing this huge PR.