Use WorkingSetBytes if memory usage is not reported #25407

brianharwell · 2021-04-28T19:53:26Z

What does this PR do?

This will use workingSetBytes for the memory usage if usageMem is zero

Why is it important?

We have Windows containers running in Kubernetes but the kubelet only reports the working set bytes for a pod and not the memory usage bytes. As a result the field kubernetes.pod.memory.usage.limit.pct is reported as 0 even though the pod has a memory limit.

This is important because without kubernetes.pod.memory.usage.limit.pct we cannot alert or monitor based on how close the pod's memory is compared to it's memory limit.

The memory usage bytes is reported for linux pods.

Here is a sample from the kubelet. Metricbeat uses this json to report pod metrics.

{
   "podRef":{
      "name":"metricbeat-node-kkl29",
      "namespace":"logging",
      "uid":"5be7cd99-712f-47f0-84e1-eba75f00f671"
   },
   "startTime":"2021-04-21T19:56:15Z",
   "containers":[
      {
         "name":"metricbeat",
         "startTime":"2021-04-21T19:56:17Z",
         "cpu":{
            "time":"2021-04-22T18:34:31Z",
            "usageNanoCores":21051518,
            "usageCoreNanoSeconds":367953125000
         },
         "memory":{
            "time":"2021-04-22T18:34:31Z",
            "workingSetBytes":196808704
         },
         "rootfs":{
            "time":"2021-04-22T18:34:31Z",
            "availableBytes":43572236288,
            "capacityBytes":106846744576,
            "usedBytes":0
         },
         "logs":{
            "time":"2021-04-22T18:34:32Z",
            "availableBytes":43572236288,
            "capacityBytes":106846744576,
            "usedBytes":0,
            "inodesUsed":0
         }
      }
   ],
   "cpu":{
      "time":"2021-04-21T19:56:17Z",
      "usageNanoCores":21051518,
      "usageCoreNanoSeconds":367953125000
   },
   "memory":{
      "time":"2021-04-22T18:34:31Z",
      "availableBytes":0,
      "usageBytes":0,
      "workingSetBytes":196808704,
      "rssBytes":0,
      "pageFaults":0,
      "majorPageFaults":0
   },
   "network":{
      "time":"2021-04-22T18:34:32Z",
      "name":"e94f1b6bc8728316148684734375542f05a93e9a5df43fa8392f08af9bf68e1b_cbr0",
      "rxBytes":302076010,
      "txBytes":70445444,
      "interfaces":[
         {
            "name":"e94f1b6bc8728316148684734375542f05a93e9a5df43fa8392f08af9bf68e1b_cbr0",
            "rxBytes":302076010,
            "txBytes":70445444
         }
      ]
   },
   "volume":[
      {
         "time":"2021-04-21T19:56:26Z",
         "availableBytes":43691851776,
         "capacityBytes":106846744576,
         "usedBytes":5229,
         "inodesFree":0,
         "inodes":0,
         "inodesUsed":0,
         "name":"config"
      },
      {
         "time":"2021-04-21T19:56:26Z",
         "availableBytes":43691851776,
         "capacityBytes":106846744576,
         "usedBytes":6050,
         "inodesFree":0,
         "inodes":0,
         "inodesUsed":0,
         "name":"metricbeat-token-dnf9s"
      }
   ],
   "ephemeral-storage":{
      "time":"2021-04-22T18:34:32Z",
      "availableBytes":43572236288,
      "capacityBytes":106846744576,
      "usedBytes":5229,
      "inodesUsed":0
   }
}

Checklist

I have zero experience with Go. I didn't want to ask someone else to make the change because this seemed rather simple.

[X ] My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Logic is correct

How to test this PR locally

Test using a Linux container and a Windows container. I can assist with both of these.

Related issues

Elastic Support Ticket #00711427

Use cases

N/A

Screenshots

Logs

cla-checker-service · 2021-04-28T19:53:29Z

❌ Author of the following commits did not sign a Contributor Agreement:
93e938c

Please, read and sign the above mentioned agreement if you want to contribute to this project

elasticmachine · 2021-04-28T20:05:28Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25407 opened
Reason: The PR is not allowed to run in the CI yet
Start Time: 2021-04-28T19:53:44.511+0000
Duration: 11 min 22 sec
Commit: 93e938c

Trends 🧪

Steps errors

Expand to view the steps failures

`Load a resource file from a shared library`

Took 0 min 0 sec . View more details on here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details on here
Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)

Log output

Expand to view the last 100 lines of log output

[2021-04-28T20:03:06.834Z]  > git merge 7916ad4d0f7b6f28d84c889546ae28d9b4ce6d84 # timeout=10
[2021-04-28T20:03:06.845Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-04-28T20:03:06.853Z]  > git config core.sparsecheckout # timeout=10
[2021-04-28T20:03:06.866Z]  > git checkout -f 93e938cbf737c9de4d19290070f9d9f182214def # timeout=15
[2021-04-28T20:03:11.605Z] Commit message: "Use WorkingSetBytes if workingSet is greater than zero"
[2021-04-28T20:03:11.606Z] First time build. Skipping changelog.
[2021-04-28T20:03:11.606Z] Cleaning workspace
[2021-04-28T20:03:11.774Z]  > git --version # timeout=10
[2021-04-28T20:03:11.777Z]  > git --version # 'git version 2.17.1'
[2021-04-28T20:03:11.780Z] fatal: bad object 6e56bc8bdb56e05b355e22744c4b93b4637fc0b5
[2021-04-28T20:03:11.608Z]  > git rev-parse --verify HEAD # timeout=10
[2021-04-28T20:03:11.611Z] Resetting working tree
[2021-04-28T20:03:11.611Z]  > git reset --hard # timeout=10
[2021-04-28T20:03:11.692Z]  > git clean -fdx # timeout=10
[2021-04-28T20:03:12.641Z] Timeout set to expire in 3 hr 0 min
[2021-04-28T20:03:12.668Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-04-28T20:03:12.898Z] [INFO] Number of builds to be searched 10
[2021-04-28T20:03:13.580Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-04-28T20:03:13.604Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25407/src/github.com/elastic/beats
[2021-04-28T20:03:13.643Z] [INFO] gitCheckout: Checkout SCM PR-25407 with default customisation from the Item.
[2021-04-28T20:03:13.680Z] [INFO] Override default checkout
[2021-04-28T20:03:13.750Z] Sleeping for 10 sec
[2021-04-28T20:03:23.779Z] The recommended git tool is: git
[2021-04-28T20:03:23.829Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-04-28T20:03:23.835Z] Wiping out workspace first.
[2021-04-28T20:03:23.844Z] Cloning the remote Git repository
[2021-04-28T20:03:23.844Z] Using shallow clone with depth 10
[2021-04-28T20:03:23.844Z] Avoid fetching tags
[2021-04-28T20:03:23.861Z] Cloning repository git@github.com:elastic/beats.git
[2021-04-28T20:03:23.889Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-25407/src/github.com/elastic/beats # timeout=10
[2021-04-28T20:03:23.895Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-04-28T20:03:23.895Z]  > git --version # timeout=10
[2021-04-28T20:03:23.898Z]  > git --version # 'git version 2.17.1'
[2021-04-28T20:03:23.899Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-04-28T20:03:23.903Z]  > git fetch --no-tags --progress -- git@github.com:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-04-28T20:03:45.411Z] Cleaning workspace
[2021-04-28T20:03:45.435Z] Using shallow fetch with depth 10
[2021-04-28T20:03:45.435Z] Pruning obsolete local branches
[2021-04-28T20:03:46.475Z] Merging remotes/origin/master commit 7916ad4d0f7b6f28d84c889546ae28d9b4ce6d84 into PR head commit 93e938cbf737c9de4d19290070f9d9f182214def
[2021-04-28T20:03:45.354Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-04-28T20:03:45.369Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-04-28T20:03:45.389Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-04-28T20:03:45.414Z]  > git rev-parse --verify HEAD # timeout=10
[2021-04-28T20:03:45.425Z] No valid HEAD. Skipping the resetting
[2021-04-28T20:03:45.426Z]  > git clean -fdx # timeout=10
[2021-04-28T20:03:45.439Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-04-28T20:03:45.439Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-04-28T20:03:45.447Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/25407/head:refs/remotes/origin/PR-25407 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-04-28T20:03:46.478Z]  > git config core.sparsecheckout # timeout=10
[2021-04-28T20:03:46.482Z]  > git checkout -f 93e938cbf737c9de4d19290070f9d9f182214def # timeout=15
[2021-04-28T20:03:48.156Z] Merge succeeded, producing 93e938cbf737c9de4d19290070f9d9f182214def
[2021-04-28T20:03:48.157Z] Checking out Revision 93e938cbf737c9de4d19290070f9d9f182214def (PR-25407)
[2021-04-28T20:03:48.405Z] Commit message: "Use WorkingSetBytes if workingSet is greater than zero"
[2021-04-28T20:03:48.405Z] Cleaning workspace
[2021-04-28T20:03:48.714Z]  > git --version # timeout=10
[2021-04-28T20:03:48.718Z]  > git --version # 'git version 2.17.1'
[2021-04-28T20:03:48.723Z] fatal: bad object 6e56bc8bdb56e05b355e22744c4b93b4637fc0b5
[2021-04-28T20:03:48.127Z]  > git remote # timeout=10
[2021-04-28T20:03:48.131Z]  > git config --get remote.origin.url # timeout=10
[2021-04-28T20:03:48.135Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-04-28T20:03:48.139Z]  > git merge 7916ad4d0f7b6f28d84c889546ae28d9b4ce6d84 # timeout=10
[2021-04-28T20:03:48.151Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-04-28T20:03:48.159Z]  > git config core.sparsecheckout # timeout=10
[2021-04-28T20:03:48.163Z]  > git checkout -f 93e938cbf737c9de4d19290070f9d9f182214def # timeout=15
[2021-04-28T20:03:48.408Z]  > git rev-parse --verify HEAD # timeout=10
[2021-04-28T20:03:48.411Z] Resetting working tree
[2021-04-28T20:03:48.411Z]  > git reset --hard # timeout=10
[2021-04-28T20:03:48.632Z]  > git clean -fdx # timeout=10
[2021-04-28T20:03:49.921Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-04-28T20:03:50.607Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-04-28T20:04:58.399Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25407/src/github.com/elastic/beats/.git
[2021-04-28T20:04:58.545Z] Archiving artifacts
[2021-04-28T20:04:59.279Z] + git rev-parse HEAD
[2021-04-28T20:04:59.617Z] + git rev-parse HEAD
[2021-04-28T20:04:59.932Z] + git rev-parse origin/pr/25407
[2021-04-28T20:04:59.989Z] [INFO] githubEnv: Found Git Build Cause: pr
[2021-04-28T20:05:00.337Z] Masking supported pattern matches of $GITHUB_TOKEN
[2021-04-28T20:05:01.909Z] [WARN] githubApiCall: The REST API call https://api.github.com/repos/elastic/beats/pulls/25407/reviews return 0 elements
[2021-04-28T20:05:01.973Z] [INFO] githubPrCheckApproved: Title: Use WorkingSetBytes if memory usage is not reported - User: brianharwell - Author Association: FIRST_TIME_CONTRIBUTOR
[2021-04-28T20:05:02.387Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet
[2021-04-28T20:05:02.387Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)
[2021-04-28T20:05:02.472Z] [INFO] Let's stop build #1. The PR is not allowed to run in the CI yet
[2021-04-28T20:05:02.494Z] Sleeping for 5 sec
[2021-04-28T20:05:03.699Z] Stage "Lint" skipped due to earlier failure(s)
[2021-04-28T20:05:03.783Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-04-28T20:05:03.866Z] Stage "Extended" skipped due to earlier failure(s)
[2021-04-28T20:05:03.953Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-28T20:05:04.033Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-04-28T20:05:04.148Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25407/src/github.com/elastic/beats
[2021-04-28T20:05:05.180Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-25407
[2021-04-28T20:05:05.441Z] [INFO] getVaultSecret: Getting secrets
[2021-04-28T20:05:05.486Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-28T20:05:07.731Z] + chmod 755 generate-build-data.sh
[2021-04-28T20:05:07.732Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25407/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25407/runs/1 ABORTED 681819
[2021-04-28T20:05:07.732Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25407/runs/1/steps/?limit=10000 -o steps-info.json
[2021-04-28T20:05:08.430Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25407/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-04-28T20:05:09.131Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-04-28T20:05:10.578Z] Retry 2/3 exited 22, retrying in 2 seconds...
[2021-04-28T20:05:12.832Z] Retry 3/3 exited 22, no more retries left.
[2021-04-28T20:05:12.832Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25407/runs/1/log/ -o pipeline-log.txt

elasticmachine · 2021-04-28T22:31:53Z

Pinging @elastic/integrations (Team:Integrations)

ycombinator · 2021-04-28T22:32:56Z

Hi @brianharwell, could you please sign the Elastic CLA using the same email address as you used for commits to this PR? That will let us then proceed with reviewing it. Thanks!

brianharwell · 2021-04-29T12:53:19Z

@ycombinator I signed it

ycombinator · 2021-04-29T12:58:53Z

@brianharwell It needs to be signed with the same email address as you use for your git commits. It looks like you signed the CLA with your gmail.com email address but your commit is authored with your paylocity.com email address.

brianharwell · 2021-04-29T13:16:21Z

@ycombinator Ok, I understand. I'll fix and resubmit

brianharwell · 2021-04-29T13:18:37Z

@ycombinator I created a new PR with correct email address here #25428

Use WorkingSetBytes if workingSet is greater than zero

93e938c

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 28, 2021

brianharwell changed the title ~~Use WorkingSetBytes if workingSet is greater than zero~~ Use WorkingSetBytes if memory usage is not reported Apr 28, 2021

ycombinator added the Team:Integrations Label for the Integrations team label Apr 28, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 28, 2021

brianharwell closed this Apr 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use WorkingSetBytes if memory usage is not reported #25407

Use WorkingSetBytes if memory usage is not reported #25407

brianharwell commented Apr 28, 2021

cla-checker-service bot commented Apr 28, 2021

elasticmachine commented Apr 28, 2021

Build stats

Trends 🧪

`Load a resource file from a shared library`

`Error signal`

elasticmachine commented Apr 28, 2021

ycombinator commented Apr 28, 2021

brianharwell commented Apr 29, 2021

ycombinator commented Apr 29, 2021

brianharwell commented Apr 29, 2021

brianharwell commented Apr 29, 2021

Use WorkingSetBytes if memory usage is not reported #25407

Use WorkingSetBytes if memory usage is not reported #25407

Conversation

brianharwell commented Apr 28, 2021

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service bot commented Apr 28, 2021

elasticmachine commented Apr 28, 2021

❕ Build Aborted

Build stats

Trends 🧪

Steps errors

Load a resource file from a shared library

Error signal

Log output

elasticmachine commented Apr 28, 2021

ycombinator commented Apr 28, 2021

brianharwell commented Apr 29, 2021

ycombinator commented Apr 29, 2021

brianharwell commented Apr 29, 2021

brianharwell commented Apr 29, 2021

`Load a resource file from a shared library`

`Error signal`