Make info logging non-blocking

[marseel]

/kind feature

Describe the solution you'd like
I would like to make it possible to make logging with info level non-blocking.
Draft idea:

• Add buffer for logging with limit of X logs that can be stored.
• If buffer is not full, add log to buffer
• If buffer is full, increment counter of skipped logs
• Every Y seconds check if some logs were skipped and log aggregated information like "Z info logs were skipped" with warning level.

This change would make info logs best effort and non-blocking.

Anything else you would like to add:
During scalability tests of kubernetes, we've observed that logging can significantly degrade performance of masters. Under high load, logging was taking up to 10s.

[marseel]

/cc @serathius

[serathius]

Having info logs as non-blocking sounds reasonable assuming that we will not change the default behavior, but introduce a flag to maintain backward compatibility
/cc @thockin @DirectXMan12 @dims
WDYT?

[dims]

@serathius i'll let the others chime in first. my question would be, is there a prior art in some other logging library?

[wojtek-t]

my question would be, is there a prior art in some other logging library?

I don't know open-source logging libraries TBH, but I know that this is something we're doing internally.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[mborsz]

/remove-lifecycle stale

[erain]

/cc @erain

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[wojtek-t]

/remove-lifecycle stale

[serathius]

Related change that introduces buffering info logs in Json logging format kubernetes/kubernetes#104873

[serathius]

@marseel Info log buffering was implemented in v1.23 kubernetes as part of JSON log format. Please check if this fulfills your requirements.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[serathius]

Closing as feature was implemented.
/close

[k8s-ci-robot]

@serathius: Closing this issue.

In response to this:

Closing as feature was implemented.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.