Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize allocation behavior of the autoscaler #10375

Merged
merged 3 commits into from
Dec 15, 2020
Merged

Optimize allocation behavior of the autoscaler #10375

merged 3 commits into from
Dec 15, 2020

Conversation

markusthoemmes
Copy link
Contributor

Proposed Changes

This optimizes the autoscaler's allocation behavior by

  1. Collapsing metric record calls, since that path is kind of expensive
  2. Avoiding to build a new logger and instead only adding the metric key where relevant
  3. Avoiding to build new loggers for the stable and panic debug logs
  4. Avoiding to call formatting logs by checking the log levels first

With the benchmark shipped in #10374, this results in the following gains:

benchmark                  old ns/op     new ns/op     delta
BenchmarkAutoscaler-16     12246         3323          -72.86%

benchmark                  old allocs     new allocs     delta
BenchmarkAutoscaler-16     83             22             -73.49%

benchmark                  old bytes     new bytes     delta
BenchmarkAutoscaler-16     6131          920           -84.99%

Pulling in knative/pkg#1964 will further reduce this to:

benchmark                  old ns/op     new ns/op     delta
BenchmarkAutoscaler-16     12246         2069          -83.10%

benchmark                  old allocs     new allocs     delta
BenchmarkAutoscaler-16     83             10             -87.95%

benchmark                  old bytes     new bytes     delta
BenchmarkAutoscaler-16     6131          632           -89.69%

/assign @vagababov

@markusthoemmes
Optimize allocation behavior of the autoscaler
c1f88ce 
This optimizes the autoscaler's allocation behavior by
1. Collapsing metric record calls, since that path is kind of expensive
2. Avoiding to build a new logger and instead only adding the metric key where relevant
3. Avoiding to build new loggers for the stable and panic debug logs
4. Avoiding to call formatting logs by checking the log levels first
@markusthoemmes
Add a simple autoscaler benchmark
cf3b5fd
@google-cla google-cla bot added the cla: yes Indicates the PR's author has signed the CLA. label Dec 15, 2020
@knative-prow-robot knative-prow-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 15, 2020
@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: markusthoemmes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow-robot knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 15, 2020
vagababov
vagababov reviewed Dec 15, 2020
View reviewed changes
Copy link
Contributor

@vagababov vagababov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just nits

pkg/autoscaler/scaling/autoscaler.go Outdated Show resolved Hide resolved
pkg/autoscaler/scaling/autoscaler.go
logger.With(zap.String("mode", "panic")).Debugf("Observed average scaling metric value: %0.3f, targeting %0.3f.",
observedPanicValue, spec.TargetValue)
if desugared.Core().Enabled(zapcore.DebugLevel) {
desugared.Debug(fmt.Sprintf("Observed average scaling metric value: %0.3f, targeting %0.3f.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth doing debug+sprintf on desugared, vs with+debugf?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the With allocates heavily as it creates a new intermittent logger.

@codecov
Copy link

codecov bot commented Dec 15, 2020 •
edited
Loading

Codecov Report

Merging #10375 (7bd619e) into master (0a24aef) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #10375      +/-   ##
==========================================
- Coverage   87.99%   87.93%   -0.06%     
==========================================
  Files         186      186              
  Lines        8720     8738      +18     
==========================================
+ Hits         7673     7684      +11     
- Misses        810      815       +5     
- Partials      237      239       +2
Impacted Files Coverage Δ
pkg/autoscaler/scaling/autoscaler.go 97.88% <100.00%> (+0.24%) ⬆️
pkg/autoscaler/statforwarder/forwarder.go 90.74% <0.00%> (-5.56%) ⬇️
pkg/autoscaler/statforwarder/processor.go 83.33% <0.00%> (-5.56%) ⬇️
pkg/autoscaler/scaling/multiscaler.go 87.93% <0.00%> (-1.73%) ⬇️
pkg/reconciler/autoscaling/kpa/scaler.go 88.57% <0.00%> (-1.43%) ⬇️
pkg/apis/serving/fieldmask.go 95.48% <0.00%> (+0.01%) ⬆️
pkg/apis/config/features.go 93.75% <0.00%> (+0.41%) ⬆️
pkg/reconciler/configuration/configuration.go 88.28% <0.00%> (+1.56%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a24aef...7bd619e. Read the comment docs.

@knative-prow-robot knative-prow-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 15, 2020
vagababov
vagababov reviewed Dec 15, 2020
View reviewed changes
Copy link
Contributor

@vagababov vagababov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@knative-prow-robot knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 15, 2020
@vagababov
Copy link
Contributor

/test pull-knative-serving-istio-stable-no-mesh

@knative-prow-robot knative-prow-robot merged commit d2a5fde into knative:master Dec 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vagababov vagababov vagababov left review comments

@taragu taragu Awaiting requested review from taragu

Assignees

@vagababov vagababov

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/autoscale cla: yes Indicates the PR's author has signed the CLA. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@markusthoemmes @knative-prow-robot @vagababov