Perfmon metricset improvements #26886

narph · 2021-07-14T14:03:38Z

What does this PR do?

adds optional counter refresh perfmon.refresh_wildcard_counters, default is disabled now
implements PdhCollectQueryDataEx instead of waiting for at least 1 sec and calling PdhCollectQueryData again. With PdhCollectQueryDataEx we can retrieve the values after they are available.
PdhExpandWildCardPath not expanding the wildcard queries from time to time, although the call should be executed twice there was necessary a wait time between the calls, this will block the rest of the calls.
Instead, a check is done if the query has been expanded, if not, the api is called again.

Why is it important?

inconsistency in data retrieval without any exceptions

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Event collection looks more consistent, also not a rise in CPU usage or thread/handle count.

Related issues

Closes Windows - perfmon - some performance counters do not retrieve any values if there are too many pdh calls #26356

elasticmachine · 2021-07-14T14:09:13Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-05T08:30:41.298+0000
Duration: 80 min 6 sec
Commit: a9ef8a8

Test stats 🧪

Test	Results
Failed	0
Passed	9037
Skipped	2409
Total	11446

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9037
Skipped	2409
Total	11446

elasticmachine · 2021-07-14T14:18:58Z

Pinging @elastic/integrations (Team:Platforms)

exekias

Thanks for working on this! left a few comments & questions

metricbeat/helper/windows/pdh/pdh_query_windows.go

metricbeat/module/windows/perfmon/reader.go

mergify · 2021-07-20T07:53:17Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b perfmon-update upstream/perfmon-update
git merge upstream/master
git push upstream perfmon-update

CHANGELOG.next.asciidoc

exekias · 2021-07-26T15:16:17Z

metricbeat/helper/windows/pdh/pdh_query_windows.go

@@ -218,6 +234,7 @@ func (q *Query) ExpandWildCardPath(wildCardPath string) ([]string, error) {
 		if err == PDH_MORE_DATA {
 			expdPaths, err = PdhExpandCounterPath(utfPath)
 		}
+		expdPaths, err = PdhExpandCounterPath(utfPath)


is this intended? IT would look like you would call PdhExpandCounterPath many times here (both in case of error and no error)

it was not intended, just for testing purposes, refactored the func

mergify · 2021-07-27T09:37:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b perfmon-update upstream/perfmon-update
git merge upstream/master
git push upstream perfmon-update

mergify · 2021-08-03T09:08:28Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b perfmon-update upstream/perfmon-update
git merge upstream/master
git push upstream perfmon-update

exekias · 2021-08-05T08:19:31Z

CHANGELOG.next.asciidoc

+- Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service {pull}22874[22874]
+- Fix reporting of cgroup metrics when running under Docker {pull}22879[22879]
+- Fix typo in config docs {pull}23185[23185]
+- Add FAQ entry for madvdontneed variable {pull}23429[23429]
+- Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete {pull}23419[23419]
+- Fix error loop with runaway CPU use when the Kafka output encounters some connection errors {pull}23484[23484]
+- Fix issue discovering docker containers and metadata after reconnections {pull}24318[24318]
+- Fix ILM alias creation when write alias exists and initial index does not exist {pull}26143[26143]
+- Omit full index template from errors that occur while loading the template. {pull}25743[25743]
+- In the script processor, the `decode_xml` and `decode_xml_wineventlog` processors are now available as `DecodeXML` and `DecodeXMLWineventlog` respectively.
+- Fix encoding errors when using the disk queue on nested data with multi-byte characters {pull}26484[26484]
+- Improve `perfmon` metricset performance. {pull}26886[26886]


something went wrong with the last merge

fixed it, thanks

* work on perfmon improvements * changelog * rever changes * complete checks * review action * fmt * clean up * docs * fix tests * update * fix tests * fix changelog (cherry picked from commit 12c8dae)

* Perfmon metricset improvements (#26886) * work on perfmon improvements * changelog * rever changes * complete checks * review action * fmt * clean up * docs * fix tests * update * fix tests * fix changelog (cherry picked from commit 12c8dae) * fix changelog

work on perfmon improvements

9f15d68

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 14, 2021

narph added 2 commits July 14, 2021 16:05

changelog

4837a57

rever changes

639d7e1

complete checks

7522834

narph self-assigned this Jul 14, 2021

narph added the Team:Platforms Label for the Integrations - Platforms team label Jul 14, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 14, 2021

narph requested a review from exekias July 14, 2021 14:19

exekias reviewed Jul 14, 2021

View reviewed changes

review action

0f0ca1d

exekias reviewed Jul 14, 2021

View reviewed changes

metricbeat/module/windows/perfmon/reader.go Outdated Show resolved Hide resolved

fmt

a7bb99d

Merge branch 'master' into perfmon-update

3a29bce

narph requested a review from exekias July 26, 2021 12:48

exekias reviewed Jul 26, 2021

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

exekias reviewed Jul 26, 2021

View reviewed changes

narph added 6 commits July 28, 2021 13:41

clean up

1cfe543

Merge branch 'master' into perfmon-update

f1448dd

docs

7583f85

fix tests

7cf69f5

update

7581155

fix tests

daa9f85

exekias approved these changes Jul 29, 2021

View reviewed changes

Merge branch 'master' into perfmon-update

927f383

exekias reviewed Aug 5, 2021

View reviewed changes

fix changelog

a9ef8a8

narph merged commit 12c8dae into elastic:master Aug 5, 2021

narph deleted the perfmon-update branch August 5, 2021 09:58

narph mentioned this pull request Aug 5, 2021

Cherry-pick #26886 to 7.x: Perfmon metricset improvements #27254

Merged

6 tasks

narph added the v7.15.0 label Aug 5, 2021

aspacca mentioned this pull request Aug 11, 2021

Flaky test TestAddCounterInvalidArgWhenQueryClosed #27312

Closed

narph mentioned this pull request Aug 22, 2022

Changed collection method of the second counter value in the Windows Perfmon module of Metricbeat #32305

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Perfmon metricset improvements #26886

Perfmon metricset improvements #26886

narph commented Jul 14, 2021 •

edited

Loading

elasticmachine commented Jul 14, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Jul 14, 2021

exekias left a comment

mergify bot commented Jul 20, 2021

exekias Jul 26, 2021

narph Jul 28, 2021

mergify bot commented Jul 27, 2021

mergify bot commented Aug 3, 2021

exekias Aug 5, 2021

narph Aug 5, 2021

Perfmon metricset improvements #26886

Perfmon metricset improvements #26886

Conversation

narph commented Jul 14, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Jul 14, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Jul 14, 2021

exekias left a comment

Choose a reason for hiding this comment

mergify bot commented Jul 20, 2021

exekias Jul 26, 2021

Choose a reason for hiding this comment

narph Jul 28, 2021

Choose a reason for hiding this comment

mergify bot commented Jul 27, 2021

mergify bot commented Aug 3, 2021

exekias Aug 5, 2021

Choose a reason for hiding this comment

narph Aug 5, 2021

Choose a reason for hiding this comment

narph commented Jul 14, 2021 •

edited

Loading

elasticmachine commented Jul 14, 2021 •

edited by jenkins-beats-ci bot

Loading