Skip to content

Commit

Permalink
Introduce InferContext (#14956)
Browse files Browse the repository at this point in the history
## Summary

I'm currently on the fence about landing the #14760 PR because it's
unclear how we'd support tracking used and unused suppression comments
in a performant way:
* Salsa adds an "untracked" dependency to every query reading
accumulated values. This has the effect that the query re-runs on every
revision. For example, a possible future query
`unused_suppression_comments(db, file)` would re-run on every
incremental change and for every file. I don't expect the operation
itself to be expensive, but it all adds up in a project with 100k+ files
* Salsa collects the accumulated values by traversing the entire query
dependency graph. It can skip over sub-graphs if it is known that they
contain no accumulated values. This makes accumulators a great tool for
when they are rare; diagnostics are a good example. Unfortunately,
suppressions are more common, and they often appear in many different
files, making the "skip over subgraphs" optimization less effective.

Because of that, I want to wait to adopt salsa accumulators for type
check diagnostics (we could start using them for other diagnostics)
until we have very specific reasons that justify regressing incremental
check performance.

This PR does a "small" refactor that brings us closer to what I have in
#14760 but without using accumulators. To emit a diagnostic, a method
needs:

* Access to the db
* Access to the currently checked file

This PR introduces a new `InferContext` that holds on to the db, the
current file, and the reported diagnostics. It replaces the
`TypeCheckDiagnosticsBuilder`. We pass the `InferContext` instead of the
`db` to methods that *might* emit diagnostics. This simplifies some of
the `Outcome` methods, which can now be called with a context instead of
a `db` and the diagnostics builder. Having the `db` and the file on a
single type like this would also be useful when using accumulators.

This PR doesn't solve the issue that the `Outcome` types feel somewhat
complicated nor that it can be annoying when you need to report a
`Diagnostic,` but you don't have access to an `InferContext` (or the
file). However, I also believe that accumulators won't solve these
problems because:

* Even with accumulators, it's necessary to have a reference to the file
that's being checked. The struggle would be to get a reference to that
file rather than getting a reference to `InferContext`.
* Users of the `HasTy` trait (e.g., a linter) don't want to bother
getting the `File` when calling `Type::return_ty` because they aren't
interested in the created diagnostics. They just want to know what
calling the current expression would return (and if it even is a
callable). This is what the different methods of `Outcome` enable today.
I can ask for the return type without needing extra data that's only
relevant for emitting a diagnostic.

A shortcoming of this approach is that it is now a bit confusing when to
pass `db` and when an `InferContext`. An option is that we'd make the
`file` on `InferContext` optional (it won't collect any diagnostics if
`None`) and change all methods on `Type` to take `InferContext` as the
first argument instead of a `db`. I'm interested in your opinion on
this.

Accumulators are definitely harder to use incorrectly because they
remove the need to merge the diagnostics explicitly and there's no risk
that we accidentally merge the diagnostics twice, resulting in
duplicated diagnostics. I still value performance more over making our
life slightly easier.
  • Loading branch information
MichaReiser authored Dec 18, 2024
1 parent ac81c72 commit 0fc4e8f
Show file tree
Hide file tree
Showing 10 changed files with 866 additions and 743 deletions.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion crates/red_knot_python_semantic/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ bitflags = { workspace = true }
camino = { workspace = true }
compact_str = { workspace = true }
countme = { workspace = true }
drop_bomb = { workspace = true }
indexmap = { workspace = true }
itertools = { workspace = true }
ordermap = { workspace = true }
Expand Down Expand Up @@ -58,4 +59,3 @@ serde = ["ruff_db/serde", "dep:serde"]

[lints]
workspace = true

50 changes: 50 additions & 0 deletions crates/red_knot_python_semantic/src/suppression.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
use salsa;

use ruff_db::{files::File, parsed::comment_ranges, source::source_text};
use ruff_index::{newtype_index, IndexVec};

use crate::{lint::LintId, Db};

#[salsa::tracked(return_ref)]
pub(crate) fn suppressions(db: &dyn Db, file: File) -> IndexVec<SuppressionIndex, Suppression> {
let comments = comment_ranges(db.upcast(), file);
let source = source_text(db.upcast(), file);

let mut suppressions = IndexVec::default();

for range in comments {
let text = &source[range];

if text.starts_with("# type: ignore") {
suppressions.push(Suppression {
target: None,
kind: SuppressionKind::TypeIgnore,
});
} else if text.starts_with("# knot: ignore") {
suppressions.push(Suppression {
target: None,
kind: SuppressionKind::KnotIgnore,
});
}
}

suppressions
}

#[newtype_index]
pub(crate) struct SuppressionIndex;

#[derive(Clone, Debug, Eq, PartialEq, Hash)]
pub(crate) struct Suppression {
target: Option<LintId>,
kind: SuppressionKind,
}

#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)]
pub(crate) enum SuppressionKind {
/// A `type: ignore` comment
TypeIgnore,

/// A `knot: ignore` comment
KnotIgnore,
}
19 changes: 9 additions & 10 deletions crates/red_knot_python_semantic/src/types.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
use std::hash::Hash;

use context::InferContext;
use diagnostic::{report_not_iterable, report_not_iterable_possibly_unbound};
use indexmap::IndexSet;
use itertools::Itertools;
use ruff_db::diagnostic::Severity;
Expand Down Expand Up @@ -29,14 +31,15 @@ use crate::stdlib::{
use crate::symbol::{Boundness, Symbol};
use crate::types::call::{CallDunderResult, CallOutcome};
use crate::types::class_base::ClassBase;
use crate::types::diagnostic::{TypeCheckDiagnosticsBuilder, INVALID_TYPE_FORM};
use crate::types::diagnostic::INVALID_TYPE_FORM;
use crate::types::mro::{Mro, MroError, MroIterator};
use crate::types::narrow::narrowing_constraint;
use crate::{Db, FxOrderSet, Module, Program, PythonVersion};

mod builder;
mod call;
mod class_base;
mod context;
mod diagnostic;
mod display;
mod infer;
Expand Down Expand Up @@ -2177,17 +2180,13 @@ pub struct InvalidTypeExpressionError<'db> {
}

impl<'db> InvalidTypeExpressionError<'db> {
fn into_fallback_type(
self,
diagnostics: &mut TypeCheckDiagnosticsBuilder,
node: &ast::Expr,
) -> Type<'db> {
fn into_fallback_type(self, context: &InferContext, node: &ast::Expr) -> Type<'db> {
let InvalidTypeExpressionError {
fallback_type,
invalid_expressions,
} = self;
for error in invalid_expressions {
diagnostics.add_lint(
context.report_lint(
&INVALID_TYPE_FORM,
node.into(),
format_args!("{}", error.reason()),
Expand Down Expand Up @@ -2827,20 +2826,20 @@ enum IterationOutcome<'db> {
impl<'db> IterationOutcome<'db> {
fn unwrap_with_diagnostic(
self,
context: &InferContext<'db>,
iterable_node: ast::AnyNodeRef,
diagnostics: &mut TypeCheckDiagnosticsBuilder<'db>,
) -> Type<'db> {
match self {
Self::Iterable { element_ty } => element_ty,
Self::NotIterable { not_iterable_ty } => {
diagnostics.add_not_iterable(iterable_node, not_iterable_ty);
report_not_iterable(context, iterable_node, not_iterable_ty);
Type::Unknown
}
Self::PossiblyUnboundDunderIter {
iterable_ty,
element_ty,
} => {
diagnostics.add_not_iterable_possibly_unbound(iterable_node, iterable_ty);
report_not_iterable_possibly_unbound(context, iterable_node, iterable_ty);
element_ty
}
}
Expand Down
49 changes: 25 additions & 24 deletions crates/red_knot_python_semantic/src/types/call.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
use super::diagnostic::{TypeCheckDiagnosticsBuilder, CALL_NON_CALLABLE};
use super::context::InferContext;
use super::diagnostic::CALL_NON_CALLABLE;
use super::{Severity, Type, TypeArrayDisplay, UnionBuilder};
use crate::Db;
use ruff_db::diagnostic::DiagnosticId;
Expand Down Expand Up @@ -86,24 +87,23 @@ impl<'db> CallOutcome<'db> {
}

/// Get the return type of the call, emitting default diagnostics if needed.
pub(super) fn unwrap_with_diagnostic<'a>(
pub(super) fn unwrap_with_diagnostic(
&self,
db: &'db dyn Db,
context: &InferContext<'db>,
node: ast::AnyNodeRef,
diagnostics: &'a mut TypeCheckDiagnosticsBuilder<'db>,
) -> Type<'db> {
match self.return_ty_result(db, node, diagnostics) {
match self.return_ty_result(context, node) {
Ok(return_ty) => return_ty,
Err(NotCallableError::Type {
not_callable_ty,
return_ty,
}) => {
diagnostics.add_lint(
context.report_lint(
&CALL_NON_CALLABLE,
node,
format_args!(
"Object of type `{}` is not callable",
not_callable_ty.display(db)
not_callable_ty.display(context.db())
),
);
return_ty
Expand All @@ -113,13 +113,13 @@ impl<'db> CallOutcome<'db> {
called_ty,
return_ty,
}) => {
diagnostics.add_lint(
context.report_lint(
&CALL_NON_CALLABLE,
node,
format_args!(
"Object of type `{}` is not callable (due to union element `{}`)",
called_ty.display(db),
not_callable_ty.display(db),
called_ty.display(context.db()),
not_callable_ty.display(context.db()),
),
);
return_ty
Expand All @@ -129,13 +129,13 @@ impl<'db> CallOutcome<'db> {
called_ty,
return_ty,
}) => {
diagnostics.add_lint(
context.report_lint(
&CALL_NON_CALLABLE,
node,
format_args!(
"Object of type `{}` is not callable (due to union elements {})",
called_ty.display(db),
not_callable_tys.display(db),
called_ty.display(context.db()),
not_callable_tys.display(context.db()),
),
);
return_ty
Expand All @@ -144,12 +144,12 @@ impl<'db> CallOutcome<'db> {
callable_ty: called_ty,
return_ty,
}) => {
diagnostics.add_lint(
context.report_lint(
&CALL_NON_CALLABLE,
node,
format_args!(
"Object of type `{}` is not callable (possibly unbound `__call__` method)",
called_ty.display(db)
called_ty.display(context.db())
),
);
return_ty
Expand All @@ -158,23 +158,22 @@ impl<'db> CallOutcome<'db> {
}

/// Get the return type of the call as a result.
pub(super) fn return_ty_result<'a>(
pub(super) fn return_ty_result(
&self,
db: &'db dyn Db,
context: &InferContext<'db>,
node: ast::AnyNodeRef,
diagnostics: &'a mut TypeCheckDiagnosticsBuilder<'db>,
) -> Result<Type<'db>, NotCallableError<'db>> {
match self {
Self::Callable { return_ty } => Ok(*return_ty),
Self::RevealType {
return_ty,
revealed_ty,
} => {
diagnostics.add(
context.report_diagnostic(
node,
DiagnosticId::RevealedType,
Severity::Info,
format_args!("Revealed type is `{}`", revealed_ty.display(db)),
format_args!("Revealed type is `{}`", revealed_ty.display(context.db())),
);
Ok(*return_ty)
}
Expand All @@ -187,14 +186,16 @@ impl<'db> CallOutcome<'db> {
call_outcome,
} => Err(NotCallableError::PossiblyUnboundDunderCall {
callable_ty: *called_ty,
return_ty: call_outcome.return_ty(db).unwrap_or(Type::Unknown),
return_ty: call_outcome
.return_ty(context.db())
.unwrap_or(Type::Unknown),
}),
Self::Union {
outcomes,
called_ty,
} => {
let mut not_callable = vec![];
let mut union_builder = UnionBuilder::new(db);
let mut union_builder = UnionBuilder::new(context.db());
let mut revealed = false;
for outcome in outcomes {
let return_ty = match outcome {
Expand All @@ -210,10 +211,10 @@ impl<'db> CallOutcome<'db> {
*return_ty
} else {
revealed = true;
outcome.unwrap_with_diagnostic(db, node, diagnostics)
outcome.unwrap_with_diagnostic(context, node)
}
}
_ => outcome.unwrap_with_diagnostic(db, node, diagnostics),
_ => outcome.unwrap_with_diagnostic(context, node),
};
union_builder = union_builder.add(return_ty);
}
Expand Down
Loading

0 comments on commit 0fc4e8f

Please sign in to comment.