Proposal: individualized context for distinct modules

[lcwangchao]

Currently, all modules share a same context named sessionctx.Context, and it brings some problems such as cycle dependency and unnecessary coupling. It's better to have individualized context for distinct modules. With this change, we can improve the maintainability of the codebase with the advantages below:

• Each context is only responsible for its own module, and we do not need to have a context with logics from all modules. So it is easier to understand the code.
• We can reduce the coupling between modules and avoid cycle dependencies.
• We can easily add new fields to the context of a module without affecting other modules.

For example, we can define a new interface EvalContext to provide information for expression evaluations:

type EvalContext interface {
	BaseEvalInfo() BaseEvalInfo
}

And then we can define a new context evalContext to implement it

// evalContext implements EvalContext
type evalContext struct {
	sctx sessionctx.Context
}

func (ctx evalContext) BaseEvalInfo() BaseEvalInfo {
    return BaseEvalInfo{
		TypeCtx: ctx.sctx.GetSessionVars().StmtCtx.TypeCtx(),
		ErrCtx: ctx.sctx.GetSessionVars().StmtCtx.ErrCtx(),
		SQLMode: ctx.sctx.GetSessionVars().SQLMode,
    }
}

Different with the previous design, the above example just uses sessionctx.Context to provide some plain "state" instead of providing "semantics." EvalContext which is a wrapper of sessionctx.Context is the one that provides "semantics" for expression evaluations. In this way, different modules can depend on the packages they really need, and we can avoid cycle dependencies. The design of distinct contexts also brings flexibility because each module can have their own assumptions about the context without putting them together into a single context.

Some New Contexts to Introduce

Some important new contexts are listed below:

EvalContext

As we mentioned above, EvalContext provides information for expression evaluations:

type EvalContext interface {
	BaseEvalInfo() BaseEvalInfo
	Level() EvalContextLevel
	FreezeToLevel(level EvalContextLevel) (EvalContext, error)
}

Only BaseEvalInfo is required for now to get the basic information for evaluating expressions. BaseEvalInfo is defined as a immutable struct:

type BaseEvalInfo struct {
    typeCtx types.Context
    errCtx erctx.Context
    sqlMode mysql.SQLMode
    ...
}

For most expression evaluations, BaseEvalInfo is enough. However, for some other special functions, extra information is required. For example, tidb_is_ddl_owner needs to know whether the current TiDB node is a DDL owner that is not included in BaseEvalInfo, so in this case the input EvalContext should be cast to another interface:

type ExtenedEvalContext struct {
    EvalContext
    IsDDLOwner() bool
    ...
}

Some extra information can be added to ExtenedEvalContext.

Providing multiple interfaces for eval context is that not cases need all information in an evaluation process. For example, if we want to evaluate an expression from a generated column, BaseEvalInfo is enough. Make an easy way to build EvalContext is helpful for some scenes that we cannot get a sessionctx.Context directly, such as the scene of lighting import.

Since we have multiple interfaces for EvalContext, the method Level returns which level of information the current context can provide. It's return value is EvalContextLevel and is defined as below:

type EvalContextLevel int

const (
	EvalContextLevelBase EvalContextLevel = iota
	EvalContextLevelExtened
	EvalContextLevelFull
)

• EvalContextLevelBase means the context only provides BaseEvalInfo. It is simple and efficient to build and copy. It is always used in some simple cases, such as evaluating an expression from a generated column.
• EvalContextLevelBase means the context provides some more extended info. This type of context is not easy to build, it always reads the information from a inner sessionctx.Context. But we can copy it from to a new one to detach it from the inner sessionctx.Context.
• EvalContextLevelFull provides the full context, all expressions can use it as the input. However, it cannot be detached from the inner sessionctx.Context because it should provide some complex states that is hard to copy.

Freeze is a special method that returns a new EvalContext with the specified level. A "frozen" context is not only thread safe, it's inner state should be copied from the previous context and can keep immutable even if the inner sessionctx.Context is changed. This is useful for some cases, such as cursor fetch mode. A context with a higher level can be frozen to a lower level, but a context with a lower level cannot be frozen to a higher level. For example, a context with level EvalContextLevelFull can be frozen to EvalContextLevelExtened or EvalContextLevelBase, but a context with level EvalContextLevelBase cannot be frozen to EvalContextLevelExtened. Freeze a context with any level to EvalContextLevelFull is also not allowed, because this level is not able to be detached from the inner sessionctx.Context.

We should also add a new method EvalContextRequirement to expression.Expression to indicate which level of EvalContext is required for the expression evaluation:

type Expression interface {
	...
	EvalContextRequirement() EvalContextLevel
	Eval(ctx EvalContext, row chunk.Row) (types.Datum, error)
	...
}

EvalContextRequirement is useful in some scenes such as cursor fetches. If a statement includes an expression which requires a eval context with level EvalContextLevelBase or EvalContextLevelExtened, that means we can fetch the data lazily in cursor fetch mode by cloning the context ignoring whether another new statement is created. Otherwise, we should fall back to
the previous implement to fetch all data before returning the cursor.

ExprBuildContext

ExprBuildContext is a context for building expressions:

type ExprBuildContext interface {
    EvalContext
    // some other methods
    ...
}

In practice, the procedure of building an expression always needs to do some evaluations, so it also extends the interface EvalContext.

Some methods will replace ExprBuildContext with sessionctx.Context as the input argument:

type functionClass interface {
    // getFunction gets a function signature by the types and the counts of given arguments.
-    getFunction(ctx sessionctx.Context, args []Expression) (builtinFunc, error)
+    getFunction(ctx ExprBuildContext, args []Expression) (builtinFunc, error)
    // verifyArgsByCount verifies the count of parameters.
    verifyArgsByCount(l int) error
}

// NewFunction creates a new scalar function or constant via a constant folding.
- func NewFunction(ctx sessionctx.Context, funcName string, retType *types.FieldType, args ...Expression) (Expression, error) {
+ func NewFunction(ctx ExprBuildContext, funcName string, retType *types.FieldType, args ...Expression) (Expression, error) {
	return newFunctionImpl(ctx, 1, funcName, retType, defaultScalarFunctionCheck, args...)
}

PlanContext

PlanContext is a context for building and optimizing plans:

type PlanContext interface {
    EvalContext
    // some other methods
    ...
}

PlanContext also extends EvalContext because it needs to do some evaluations in building and optimizing procedure.

Some methods will replace PlanContext with sessionctx.Context as the input argument or return value:

type Plan interface {
    ...
-    SCtx() sessionctx.Context
+    Ctx() PlanContext
    ...
}

- func Optimize(ctx context.Context, sctx sessionctx.Context, node ast.Node, is infoschema.InfoSchema) (plan core.Plan, slice types.NameSlice, retErr error) {
+ func Optimize(ctx context.Context, pctx core.PlanContext, node ast.Node, is infoschema.InfoSchema) (plan core.Plan, slice types.NameSlice, retErr error) {
    ...
}

table.MutateContext & table.AllocatorContext

These two contexts are used to provide information for methods in interface tables.Table.

table.MutateContext is defined as:

type MutateContext interface {
	expression.BuildContext
        // ... some other methods
}

It is used to provide context when we are mutating a table and should extend expression.BuildContext to build expressions defined in table.

AllocatorContext is defined as:

type AllocatorContext interface {
	// GetTemporaryTable returns some runtime information for temporary tables.
	GetTemporaryTable(tbl *model.TableInfo) tableutil.TempTable
}

It only contains one method GetTemporaryTable to provide some runtime information for the temporary table to allocate id in session level instead of global level.

ExecutorContext

ExecutorContext is a context for executor executing:

type ExecutorContext interface {
    EvalContext
    // some other methods
    ...
}

ExecutorContext also extends EvalContext because it needs to do some evaluations in the executing procedure. The type of ctx inner BaseExecutor will be replaced as ExecutorContext:

type BaseExecutor struct {
-    ctx sessionctx.Context
+    ctx ExecutorContext
}

Tasks

Expression

• introduce EvalContext to provide context in expression evaluation #49415
• introduce expression.BuildContext to build expressions #50661
• Make context more simple for table.Index #50691
• Introduce table.MutateContext and table.AllocatorContext context #50861
• introduce PlanContext to provide context for planner #51070
• make sessionctx.Context depend on contexts in planner and expression #51149
• replace context in planbuilder to PlanContext #51171
• introduce AggFuncUpdateContext and AggFuncBuildContext to evaluate and build aggeration functions #51223
• support to implement PlanContext in separate package outside session #51236
• support to implement table.MutateContext to separate package outside session #51259
• support to implement expression.EvalContext in separate packages outside sessionctx #51298
• seperate methods in PlanContext and EvalContext/BuildContext #51381
• expose some basic fields to expression.EvalContext #51419
• "optional" properties for expression.EvalContext #51477
• EvalContext should not provide property EnableVectorizedExpression #51563
• force scalar expression to implement RequireOptionalEvalProps explicitly #51656
• Add expression.PushDownContext to provide information for context push down #51890

[YangKeao]

Now, we have introduced a lot of contexts. I tried to illustrate the relationship between them in a graphics:

The green color means (at least on implementation of) it is simple and easy to copy/clone. The red color means it's not safe/easy to copy/clone, usually means it still depends on session context.