cmd/compile: improve inlining cost model

[josharian]

The current inlining cost model is simplistic. Every gc.Node in a function has a cost of one. However, the actual impact of each node varies. Some nodes (OKEY) are placeholders never generate any code. Some nodes (OAPPEND) generate lots of code.

In addition to leading to bad inlining decisions, this design means that any refactoring that changes the AST structure can have unexpected and significant impact on compiled code. See CL 31674 for an example.

Inlining occurs near the beginning of compilation, which makes good predictions hard. For example, new or make or & might allocate (large runtime call, much code generated) or not (near zero code generated). As another example, code guarded by if false still gets counted. As another example, we don't know whether bounds checks (which generate lots of code) will be eliminated or not.

One approach is to hand-write a better cost model: append is very expensive, things that might end up in a runtime call are moderately expensive, pure structure and variable/const declarations are cheap or free.

Another approach is to compile lots of code and generate a simple machine-built model (e.g. linear regression) from it.

I have tried both of these approaches, and believe both of them to be improvements, but did not mail either of them, for two reasons:

• Inlining significantly impacts binary size, runtime performance, and compilation speed. Minor changes to the cost model or to the budget can have big impacts on all three. I hoped to find a model and budget that was clearly pareto optimal, but I did not. In order to make forward progress, we need to make a decision about what metric(s) we want to optimize for, and which code to measure those metric on. This is to my mind the single biggest blocker for improving inlining.
• Changing inlining decisions impacts the runtime as well as other code and minor inlining changes to the runtime can have outsized performance impacts. I see several possible ways to address this. (1) We could add a //go:inline annotation for use in the runtime only, to allow runtime authors to force the compiler to inline performance-critical functions. If a non-inlinable function was marked //go:inline, compilation would fail. (2) We could add a //go:mustinline annotation for use in the runtime only (see CL 22785), to allow runtime authors to protect currently-inlined functions against becoming non-inlined. (3) We could tune runtime inlining (cost model, budget, etc.) independently.

Three other related ideas:

• We might want to take into account the number and size of parameters and return values of a function when deciding whether to inline it, since those determine the cost in binary size of setting up the function call.
• We might want to have separate budgets for expressions and control flow, since branches end up being more expensive on most metrics.
• We could treat intra-package inlining different than inter-package inlining. When inlining across packages, we don't actually need to decide early on whether to allow inlining, since the actual inlining will occur in an entirely different compiler invocation. We could thus wait until function compilation is complete, and we know exactly how large the fully optimized code is, and then make our decision about whether other packages should inline that function. Downsides to this otherwise appealing idea are: (1) unexpected performance impact of moving a function from one package to another, (2) it introduces a significant dependency between full compilation and writing export data, which e.g. would prevent cmd/go: build dependent packages as soon as export data is ready #15734.

cc @dr2chase @randall77 @ianlancetaylor @mdempsky

[ianlancetaylor]

I'm going to toss in a few more ideas to consider.

An unexported function that is only called once is often cheaper to inline.

Functions that include tests of parameter values can be cheaper to inline for specific calls that pass constant arguments for those parameters. That is, the cost of inlining is not solely determined by the function itself, it is also determined by the nature of the call.

Functions that only make function calls in error cases, which is fairly common, can be cheaper to handle as a mix of inlining and outlining: you inline the main control flow but leave the error handling in a separate function. This may be particularly worth considering when inlining across packages, as the export data only needs to include the main control flow. (Error cases are detectable as the control flow blocks that return a non-nil value for a single result parameter of type error.)

One of the most important optimizations for large programs is feedback directed optimization aka profiled guided optimization. One of the most important lessons to learn from feedback/profiling is which functions are worth inlining, both on a per-call basis and on a "most calls pass X as argument N" basis. Therefore, while we have no FDO/PGO framework at present, any work on inlining should consider how to incorporate information gleaned from such a framework when it exists.

Pareto optimal is a nice goal but I suspect it is somewhat unrealistic. It's almost always possible to find a horrible decision made by any specific algorithm, but the algorithm can still be better on realistic benchmarks.

[rasky]

Functions that include tests of parameter values can be cheaper to inline for specific calls that pass constant arguments for those parameters. That is, the cost of inlining is not solely determined by the function itself, it is also determined by the nature of the call.

A common case where this would apply is when calling marshallers/unmarshallers that use reflect to inspect interface{} parameters. Many of them tend to have a "fast-path" in case reflect.Type is some basic type or has some basic property. Inlining that fast-path would make it even faster. Eg: see binary.Read and think how much of its code could be pulled when the value bound to the interface{} argument is known at compile-time.

[thanm]

Along the lines of what iant@ said, it's common for C++ compilers take into account whether a callsite appears in a loop (and thus might be "hotter"). This can help for toolchains that don't support FDO/PGO or for applications in which FDO/PGO are not being used.

[minux]

No pragmas that mandates inline, please. I already expressed dislike for //go:noinline, and I will firmly object any proposal for //go:mustinline or something like that, even if it's limited to runtime. If we can't find a good heuristics for the runtime package, I don't think it will handle real-world cases well. Also, we need to somehow fix the traceback for inlined non-leaf functions first. Another idea for the inlining decision is how simpler could the function body be if inlined. Esp. for reflect using functions that has fast paths, if the input type matches the fast path, even though the function might be very complicated, the inlined version might be really simple.

[dr2chase]

Couldn't we obtain a minor improvement in the cost model by measuring the size of generated assembly language? It would require preserving a copy of the tree till after compilation, and doing compilation bottom-up (same way as inlining is scheduled) but that would give you a more accurate measure. There's a moderate chance of being able to determine goodness of constant parameters at the SSA-level, too.

Note that this would require rearranging all of these transformations (inlining, escape analysis, closure conversion, compilation) to run them function/recursive-function-nest at-a-time, so that the results from compiling bottom-most functions all the way to assembly language would be available to inform inlining at the next level up.

[josharian]

doing compilation bottom-up

I have also considered this. There'd be a lot of high risk work rearranging the rest of the compiler to work this way. It could also hurt our chances to get a big boost out of concurrent compilation; you want to start on the biggest, slowest functions ASAP, but those are the most likely to depend on many other functions.

[dr2chase]

It doesn't look that high risk to me; it's just another iteration order. SSA also gives us a slightly more tractable place to compute things like "constant parameter values that shrink code size", even if it is only so crude as looking for blocks directly conditional on comparisons with parameter values.

[josharian]

I think we could test the inlining benefits of the bottom-up compilation pretty easily. One way is to do it just for inter-package compilation (as suggested above); another is to hack cmd/compile to dump the function asm size somewhere and then hack cmd/go to compile all packages twice, using the dumped sizes for the second round.

[CAFxX]

An unexported function that is only called once is often cheaper to inline.

Out of curiosity, why "often"? I can't think off the top of my head a case in which the contrary is true.

Also, just to understand, in -buildmode=exe basically every single function beside the entry function is going to be considered unexported?

[ianlancetaylor]

An unexported function that is only called once is often cheaper to inline.

Out of curiosity, why "often"? I can't think off the top of my head a case in which the contrary is true.

It is not true when the code looks like

func internal() {
    // Large complex function with loops.
}

func External() {
    if veryRareCase {
        internal()
    }
}

Because in the normal case where you don't need to call internal you don't have set up a stack frame.

Also, just to understand, in -buildmode=exe basically every single function beside the entry function is going to be considered unexported?

In package main, yes.

[CAFxX]

Because in the normal case where you don't need to call internal you don't have set up a stack frame.

Oh I see, makes sense. Would be nice (also in other cases) if setting up the stack frame could be sunk in the if, but likely it wouldn't be worth the extra effort.

Also, just to understand, in -buildmode=exe basically every single function beside the entry function is going to be considered unexported?

In package main, yes.

The tyranny of unit-at-a-time :D

[RalphCorderoy]

Functions that start with a run of if param1 == nil { return nil } where the tests and return values are simple parameters or constants would avoid the call overhead if just this part was inlined. The size of setting up the call/return could be weighed against the size of the simple tests.

[mvdan]

@RalphCorderoy I've been thinking about the same kind of function body "chunking" for early returns. Especially interesting for quick paths, where the slow path is too big to inline.

Unless the compiler chunks, it's up to the developer to split the function in two I presume.

[RalphCorderoy]

Hi @mvdan, Split the function in two with the intention the compiler then inlines the non-leaf first one?
Another possibility on stripping some of the simple quick paths is the remaining slow non-inlined function no longer needs some of the parameters.

[mvdan]

Yes, for example, here tryFastPath is likely small enough to be inlined (if forward thunk funcs were inlineable, see #8421):

func tryFastPath() {
    if someCond {
        // fast path
        return ...
    }
    return slowPath()
}

[josharian]

Too late for 1.9.

[gopherbot]

Change https://golang.org/cl/57410 mentions this issue: runtime: add TestIntendedInlining

[TocarIP]

Another example of current inlining heuristic punishing more readable code

if !foo {
  return
}
if !bar {
  return
}
if !baz {
  return
}
//do stuff

is more "expensive" than

if foo && bar && baz {
// do stuff
}
return

This is based on real code from regexp package (see https://go-review.googlesource.com/c/go/+/65491
for details)

[ghasemloo]

related: https://lemire.me/blog/2017/09/05/go-does-not-inline-functions-when-it-should/

[josharian]

Here’s a compromise idea that I imagine will please no one. :)

Allow call site annotations like /*go1.16:inline*/. (Maybe also function-site annotations?) /* annotations are necessary because there may be multiple calls on one line. For versions other than go 1.16, this is a no-op. And the annotation is a no-op unless you pass the compiler flag -l=manual. That compiler flag disables inlining except exactly where annotated. (Or maybe the existence of a single relevant annotation disables all inlining except exactly that requested.) If a requested inlining at a call site is impossible, compilation fails.

This grants complete control to the user who really needs it. It is enough of a pain to discourage casual use. It is scoped to exactly one Go version to avoid accidental rot. It provides raw data about optimal inlining decisions to compiler authors.

[randall77]

Not sure I see the argument here. Im not arguing for function annotation, I explicitly argued for a callsite annotation.

I'm not specifically talking about your suggestion. Just the general idea of inline annotations, both callsite and declaration site.

Why wouldn't be callsite annotations be enough for library functions?

func (t *T) Foo() int {
    return t.x + t.y
}

With callsite annotations every place t.Foo is used needs an annotation. As opposed to just once at the definition. Which will lead to library authors adding "This function will perform better if you add a @inline annotation at each call site" to their docs.

Hence why I specifically argue about an annotation on the callsite and not on the function. I agree library authors sometime feel the pressure you're mentioning, but as I clearly stated manual inlining is happening today for that very reason, and therefore I find it hard to consider this as a very compelling argument against the callsite annotation.

This will happen with both callsite or declaration annotations, just at a slightly different level.

func Foo() {
   internalFoo()
}

If the library containing Foo isn't a hot path in my application, I'd rather not have Foo inlined into its call sites, or have internalFoo inlined into Foo. So an inline annotation at the declaration of Foo and/or one at the callsite of internalFoo are both at least unnecessary and likely code-size harmful (assuming that the bodies were big enough to not be covered by the inlining heuristic we have today).

[mvdan]

Quick observation: right now, the manual inlining generally only happens within a module or even a single package, not crossing module boundaries. Perhaps some form of directive could have a similar restriction, to prevent public APIs from being littered with dubious performance directives. Or perhaps we could even restrict it to unexported functions.

[zeebo]

I was seconds away from posting a similar suggestion about only allowing inlining annotations on unexported functions.

That would cover every case I've ever ran into and avoid all of the terrible manual inlining I have done for substantial gains over the literal years.

---

edit: The main pain with not having control comes from exploratory programming. It is a ton of work to manually inline, find out it's not a great idea, and then undo. Then do that a ton over many refactorings that could change the assumptions. I would personally be happy manually inlining still if it wasn't so painful to discover when it was a good idea for a specific case. So even if the annotations were ignored unless compiled with a special flag, I'd still be happy.

[CAFxX]

I'm not specifically talking about your suggestion. Just the general idea of inline annotations, both callsite and declaration site.

I was just pointing out that the arguments you made don't really apply to callsite annotations as much as they apply to function annotations, though. Conflating the two don't seem to help discussion.

With callsite annotations every place t.Foo is used needs an annotation.

Each place where the caller cares enough to and the inliner doesn't do it automatically, yes. That is pretty much exactly what we want to avoid the copy-paste scenario. Why would that be an argument against callsite annotations? We all agree it would be ideal if the inliner was better so that we didn't have to; the assumption is that it likely won't become so much better that there will be no need for manual control.

As opposed to just once at the definition. Which will lead to library authors adding "This function will perform better if you add a @inline annotation at each call site" to their docs.

That is pretty much what is already happening today, with the difference right now the inlining has to be done manually. Callsite annotations wouldn't make this worse, on the contrary, not only it would avoid having to duplicate code, it would potetntially even avoid having to expose additional APIs (like the one linked above).

This will happen with both callsite or declaration annotations, just at a slightly different level.

My point was that it is already happening today, with the difference that right now people have to manually inline code. That point still applies.

Furthermore, I think we agree it would happen less in the case of callsite annotations.

func Foo() {
   internalFoo()
}

If the library containing Foo isn't a hot path in my application, I'd rather not have Foo inlined into its call sites, or have internalFoo inlined into Foo. So an inline annotation at the declaration of Foo and/or one at the callsite of internalFoo are both at least unnecessary and likely code-size harmful (assuming that the bodies were big enough to not be covered by the inlining heuristic we have today).

This is definitely one valid argument. I kinda already offered a solution for that though ("even though it's pretty much orthogonal to the proposal itself, when the inliner gets better and starts to actually make decisions based on callsites instead of functions, it may actually transitively propagate it as a hint that the specific callpath is expected to be frequently executed").

Regardless, I wouldn't discount a practical solution just because it doesn't cover 100% of the possible use-cases (as by that standard Go itself likely wouldn't exist)

Quick observation: right now, the manual inlining generally only happens within a module or even a single package, not crossing module boundaries. Perhaps some form of directive could have a similar restriction, to prevent public APIs from being littered with dubious performance directives. Or perhaps we could even restrict it to unexported functions.

One note though: I still think there's value in allowing callsite annotations to request cross-module inlining (I gave one example just above, and can probably dig a few others up). Restricing function annotations to unexported symbols OTOH sounds like a pretty reasonable compromise.

And the annotation is a no-op unless you pass the compiler flag -l=manual.

Or maybe, only callsite annotations in the root module (the module that contains the main package) are enabled by default?

[mvdan]

One note though: I still think there's value in allowing callsite annotations to request cross-module inlining (I gave one example just above, and can probably dig a few others up).

I think the danger there is that people could add notes to their function definition godoc like "for maximum performance, annotate calls with...". We can strongly warn against that kind of thing, but it will happen anyway. That's why I think starting with unexported funcs would give a large part of the benefit at little risk, so it could be the first step for an annotation.

[renthraysk]

Just hit an annoyance with the inliner in past few days, reading and writing individual bits.

Writer.WriteBit() function is inlineable. By making 2 concessions/hacks, x argument an uint64, and reusing x for carry result.

However, ReadBit() which is using a similar strategy, is "function too complex: cost 88 exceeds budget 80"

[elichai]

I'm having a bunch of problems with bad inlining when implementing hash functions.
I had to manually inline functions, move things from function to another to please the inliner and more, at the end I increased the perf by 200%(which is huge for hashes) just by manually inlining and similar things.

I also can't use the useful:

func memset(s []byte, c byte) {
	for i := range s {
		s[i] = c
	}
}

Because the fact that it doesn't inline range loops, then if I call memset(s, 0) it will call to memset and then loop instead of just calling to runtime.memclrNoHeapPointers which it could've done if it knew that c=0

[martisch]

Because the fact that it doesn't inline range loops, then if I call memset(s, 0) it will call to memset and then loop instead of just calling to runtime.memclrNoHeapPointers which it could've done if it new that c=0

I dont think the existing range clear optimizations will trigger even if the inlining is changed.
Thats something that will not only need inlining but the range clear optimization detection to move into the ssa pass that can represent that c is a constant in the loop.

Better to write that directly for now:

func memclear(s []byte) {
	for i := range s {
		s[i] = 0
	}
}

If inlining is improved it can take care of the call overhead here.

[yuzefovich]

My colleague wrote a tool that folks following this thread might find useful (it allows for adding comments that are "assertions" on the compiled code for checking whether a function (or a call-site) is inlined). We at cockroachdb began introducing these "assertions" into the codebase and are verifying them during the linter test runs.

[FiloSottile]

I just came out of a fight with the inliner and figured I would report back. I was trying to outline an allocation for a function with multiple return values, and the inliner had a lot of non-obvious opinions.

Here's the version that finally worked, with cost 72.

func (v *Point) ExtendedCoordinates() (X, Y, Z, T *field.Element) {
	var e [4]field.Element
	X, Y, Z, T = v.extendedCoordinates(&e)
	return
}

Both the more idiomatic ways to write this function have costs higher than 84.

Avoiding the named returns costs 84.

func (v *Point) ExtendedCoordinates() (X, Y, Z, T *field.Element) {
	var e [4]field.Element
	return v.extendedCoordinates(&e)
}

Making separate allocations costs 90.

func (v *Point) ExtendedCoordinates() (X, Y, Z, T *field.Element) {
	var X1, Y1, Z1, T1 field.Element
	X, Y, Z, T = v.extendedCoordinates(&X1, &Y1, &Z1, &T1)
	return
}

[gopherbot]

Change https://golang.org/cl/347732 mentions this issue: cmd/compile: boost inlining into FORs