Add metricbeat iis module #15059

narph · 2019-12-11T16:25:15Z

Related issue #13072

Reusing the pdh query (in perfmon) to retrieve the iis specific counters.
Initial implementation contains 2 metricsets:

webserver - includes aggregated metrics over the entire IIS web server (light metricset based on the perfmon metricset)
website - includes metrics from each website running
application_pool - includes metrics for each application pool

- module: iis
  metricsets:
    - webserver
    - website
    - application_pool
  enabled: true
  period: 10s

 # filter on application pool names
 # application_pool.name: []

metricbeat/module/iis/performanceCounters.go

metricbeat/helper/windows/pdh/pdh_windows.go

narph · 2020-02-19T16:11:46Z

@jsoriano , I have touched on all the feedback, let me know if I miss anything

jsoriano

Thanks! Some more comments on the latest changes.

metricbeat/helper/windows/pdh/pdh_windows.go

jsoriano · 2020-02-19T16:49:53Z

metricbeat/module/iis/application_pool/reader.go

@@ -215,7 +215,7 @@ func getApplicationPools(names []string) ([]ApplicationPool, error) {
 	return filtered, nil
 }

-// getw3wpProceses func retrieves the running w3wp process ids
+// getw3wpProceses func retrieves the running w3wp process ids. A worker process is a windows process (w3wp.exe) which runs Web applications, and is responsible for handling requests sent to a Web Server for a specific application pool.


Nit. Very long line.

metricbeat/module/iis/application_pool/_meta/data.json

jsoriano · 2020-02-19T17:04:33Z

metricbeat/module/windows/perfmon/perfmon.go

 func init() {
-	mb.Registry.MustAddMetricSet("windows", "perfmon", New)
+	mb.Registry.MustAddMetricSet("windows", metricsetName, New)


I am thinking now that as these metricsets don't need a host because they only query perfmon, we could add this:

Suggested change

mb.Registry.MustAddMetricSet("windows", metricsetName, New)

mb.Registry.MustAddMetricSet("windows", metricsetName, New,

mb.WithHostParser(parse.EmptyHostParser),

)

jsoriano · 2020-02-19T17:08:52Z

metricbeat/module/windows/perfmon/perfmon.go

+		if err != nil {
+			return nil, err
+		}
+	}


Oh ok, I would prefer if we can make it more explicit, like with an additional internal flag in the metricset.

Could it happen that we create another metricset based on perfmon where the instance label is also mandatory?

jsoriano · 2020-02-19T17:12:25Z

metricbeat/module/windows/perfmon/reader.go

 	if err != nil {
 		return nil, errors.Wrap(err, "failed formatting counter values")
 	}
+	var events []mb.Event
+	// GroupAllCountersTo config option will only apply to the webserver metricset, where counters for all instances are aggregated
+	if metricsetName != metricset && re.config.GroupAllCountersTo != "" {


Similarly here, I would prefer to don't depend on the metricset name for certain features, I would prefer this to be more explicit.
Also this generic metricset shouldn't have something specific for the webserver metricset (or any other metricset) if possible.

metricbeat/module/windows/perfmon/perfmon.go

jsoriano · 2020-02-19T17:15:20Z

metricbeat/module/iis/application_pool/reader_test.go

+	assert.NotNil(t, reader.Query.Handle)
+	assert.NotNil(t, reader.Query.Counters)
+	assert.Zero(t, len(reader.Query.Counters))
+	defer reader.close()


This comment is unaddressed (not so important in any case).

jsoriano

Thanks! It LGTM, the only thing, that we could move process.pid to the root level to follow ECS, but if it is complicated to do it now with the perfmon metricset we can leave it for a future change.

metricbeat/module/iis/application_pool/_meta/data.json

metricbeat/module/iis/application_pool/reader.go

sorantis

Looks good. Suggested more metrics for the module.

sorantis · 2020-02-25T10:43:09Z

metricbeat/module/iis/website/manifest.yml

+      measurement_label: total_connection_attempts
+      query: '\Web Service(*)\Total Connection Attempts (all instances)'
+    - instance_label: 'name'
+      measurement_label: total_get_requests


@narph I can see total_get_requests, total_post_requests, total_delete_requests.
Can we also pull total_put_requests, total_head_requests, total_trace_requests, total_options_requests

\Web Service(*)\Total PUT Requests
\Web Service(*)\PUT Requests/sec

\Web Service(*)\Total Head Requests
\Web Service(*)\Head Requests/sec

\Web Service(*)\Options Requests/sec
\Web Service(*)\Total Options Requests

\Web Service(*)\Other Request Methods/sec
\Web Service(*)\Total Other Request Methods

\Web Service(*)\Total Trace Requests
\Web Service(*)\Trace Requests/sec

\Web Service(*)\Total Unlock Requests
\Web Service(*)\Unlock Requests/sec

sorantis · 2020-02-25T11:01:47Z

metricbeat/module/iis/webserver/manifest.yml

+      query: '\Web Service(_Total)\Bytes Received/sec'
+    - instance_label: ''
+      measurement_label: network.current_connections
+      query: 'Web Service(_Total)\Current Connections'


Would be great to also have
Web Service(*)Current Anonymous Users
Web Service(*)Current NonAnonymous Users
Web Service(*)Total Anonymous Users
Web Service(*)Anonymous Users/sec
Web Service(*)Total NonAnonymous Users

sorantis · 2020-02-25T11:02:41Z

metricbeat/module/iis/webserver/manifest.yml

+      query: '\Web Service(_Total)\Delete Requests/sec'
+    - instance_label: ''
+      measurement_label: network.service_uptime
+      query: '\Web Service(_Total)\Service Uptime'


We can also add error counters:
Web Service(*)Total Not Found Errors
Web Service(*)Not Found Errors/sec
Web Service(*)Total Locked Errors
Web Service(*)Locked Errors/sec

metricbeat/module/iis/webserver/manifest.yml

metricbeat/module/iis/application_pool/reader.go

* created iis module * work in progress * iis changes * add iis module * light * work on build * work on build * build * fmt update * temp * work on website * temp * temp * manifest changes * temp * temp * work on wp * adding application pool metricset * wmi option * test * work on apppool * work on app pool * work on website metricset * work on tests * Work on website * work on website * work on website * perfmon fix * work on websie * update config * feedback * work on feedback * temp * ecs * temp * add counters (cherry picked from commit 8be4589)

kaiyan-sheng

Missing changelog here?

* Add metricbeat iis module (#15059) * created iis module * work in progress * iis changes * add iis module * light * work on build * work on build * build * fmt update * temp * work on website * temp * temp * manifest changes * temp * temp * work on wp * adding application pool metricset * wmi option * test * work on apppool * work on app pool * work on website metricset * work on tests * Work on website * work on website * work on website * perfmon fix * work on websie * update config * feedback * work on feedback * temp * ecs * temp * add counters (cherry picked from commit 8be4589) * fix file

narph added 5 commits November 19, 2018 18:28

created iis module

9500d18

work in progress

bab0cd4

Merge branch 'master' into iis-module

66dfa82

iis changes

b1fba8f

add iis module

7a8ff78

narph added [zube]: In Progress Team:Integrations Label for the Integrations team labels Dec 11, 2019

narph self-assigned this Dec 11, 2019

houndci-bot reviewed Dec 11, 2019

View reviewed changes

narph added 13 commits January 15, 2020 16:50

light

6bd81eb

Merge branch 'master' into iis-module

3b7b520

work on build

f5cb0a7

work on build

c320a53

build

9d5df82

fmt update

222648e

temp

00e1a08

work on website

258ded3

temp

5f30e5a

Merge branch 'master' into iis-module

7ab4d2d

temp

92e5873

manifest changes

f18b298

temp

b770c57

andresrc added Team:Platforms Label for the Integrations - Platforms team [zube]: Inbox [zube]: In Progress and removed [zube]: In Progress [zube]: Inbox labels Jan 27, 2020

narph added 2 commits January 28, 2020 14:28

temp

ea28cbe

work on wp

397296b

houndci-bot reviewed Jan 31, 2020

View reviewed changes

metricbeat/helper/windows/pdh/pdh_windows.go Show resolved Hide resolved

metricbeat/helper/windows/pdh/pdh_windows.go Outdated Show resolved Hide resolved

feedback

ecaa80f

narph requested a review from jsoriano February 19, 2020 16:11

jsoriano reviewed Feb 19, 2020

View reviewed changes

work on feedback

fcfd4f3

narph requested a review from jsoriano February 21, 2020 12:02

jsoriano approved these changes Feb 21, 2020

View reviewed changes

metricbeat/module/iis/application_pool/_meta/data.json Outdated Show resolved Hide resolved

narph added 3 commits February 24, 2020 13:02

temp

118f55c

Merge branch 'master' into iis-module

d7b9eb9

ecs

f8ab974

jsoriano approved these changes Feb 24, 2020

View reviewed changes

metricbeat/module/iis/application_pool/reader.go Show resolved Hide resolved

sorantis reviewed Feb 25, 2020

View reviewed changes

narph added 4 commits February 25, 2020 14:14

Merge branch 'master' into iis-module

a40569e

temp

fbb7cd5

add counters

5f1b016

Merge branch 'master' into iis-module

ac81ebe

narph merged commit 8be4589 into elastic:master Feb 28, 2020

narph deleted the iis-module branch February 28, 2020 15:55

zube bot added [zube]: Done and removed [zube]: In Review labels Feb 28, 2020

narph added the needs_backport PR is waiting to be backported to other branches. label Feb 28, 2020

narph mentioned this pull request Feb 28, 2020

Cherry-pick #15059 to 7.x: Add metricbeat iis module #16690

Merged

kaiyan-sheng reviewed Mar 2, 2020

View reviewed changes

andresrc removed the [zube]: Done label Mar 4, 2020

narph added test-plan Add this PR to be manual test plan v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 27, 2020

andresrc added the test-plan-added This PR has been added to the test plan label Mar 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metricbeat iis module #15059

Add metricbeat iis module #15059

narph commented Dec 11, 2019 •

edited

Loading

narph commented Feb 19, 2020

jsoriano left a comment

jsoriano Feb 19, 2020

jsoriano Feb 19, 2020

jsoriano Feb 19, 2020

jsoriano Feb 19, 2020

jsoriano Feb 19, 2020

jsoriano left a comment •

edited

Loading

sorantis left a comment

sorantis Feb 25, 2020

sorantis Feb 25, 2020

sorantis Feb 25, 2020

sorantis Feb 25, 2020

kaiyan-sheng left a comment

-	mb.Registry.MustAddMetricSet("windows", metricsetName, New)
+	mb.Registry.MustAddMetricSet("windows", metricsetName, New,
+	                    mb.WithHostParser(parse.EmptyHostParser),
+	)

Add metricbeat iis module #15059

Add metricbeat iis module #15059

Conversation

narph commented Dec 11, 2019 • edited Loading

narph commented Feb 19, 2020

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment • edited Loading

Choose a reason for hiding this comment

sorantis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaiyan-sheng left a comment

Choose a reason for hiding this comment

narph commented Dec 11, 2019 •

edited

Loading

jsoriano left a comment •

edited

Loading