Add IIS Filebeat module #6127

cgwrench · 2018-01-21T00:13:34Z

This pull request adds an IIS Filebeat module to parse access and error logs from the IIS web server. This contribution includes:

A fileset for parsing IIS access logs;
A fileset for parsing IIS error (HTTPERR) logs;
Test files for both filesets to verify the ingest node pipelines;
A sample Kibana dashboard.

For the IIS module access fileset, three grok patterns have been implemented for the ingest pipeline definition:

A minimal pattern that represents the default fields that are logged by a fresh install of IIS;
A pattern that matches a more verbose logging configuration that is used by IIS running in Azure;
A pattern that matches the case when IIS is configured to log all available fields.

I think this represents a complete first implementation of an IIS module. Please let me know if there is anything that I've missed or that should be changed.

There are a couple of areas that I'd love feedback on:

Since IIS only runs on Windows, what is the correct way to specify that a Filebeat module should only run on Windows? For now I've only specified paths for Windows in the module manifest.yml.
One issue that I observed is that the IIS fields don't appear to be in the index template populated when Filebeat starts up. Have I missed something in the setup/build of this new module? For configuring the Kibana dashboards I manually edited the index template.

Two additional grok patterns are added to the IIS Filebeat "access" file set in order to support configurations with more verbose logging enabled. The additional patterns match log files with all IIS log fields enabled and log files generated by IIS running in Microsoft Azure.

karmi · 2018-01-21T00:13:35Z

Hi @cgwrench, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

elasticmachine · 2018-01-21T00:13:35Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

cgwrench · 2018-01-21T00:27:36Z

Hi @karmi. Apologies for that. I've added this email address to my GitHub profile now. Let me know if there is still an issue.

ruflin · 2018-01-21T22:36:17Z

@cgwrench @ph I wonder how much this PR overlaps with #5685 as I see @cgwrench also commenting there. In any case, thanks for pushing this forward.

cgwrench · 2018-01-22T09:21:58Z

Hi @ruflin. There is some overlap between the two pull requests. I had started on my own implementation when I saw that #5685 added a basic IIS access logs fileset. My comments/review on that pull request suggested some improvements that could be made (as summarised here) that were based on my own experience working on this feature. In the absence of any response or progress on these suggestions I've carried working on and testing my own fork, which now address all of the feedback noted in #5685 (comment) and addresses a couple of bugs I found in in the changes proposed in #5685. Given that I think I've got a complete implementation of an IIS Filebeat module, I thought I'd submit this as a new pull request. I'm more than happy to handle this different though: just let me know how you'd like me to approach this.

ph

@cgwrench Thanks for carrying this over, I've made a few small comments on the PR.

One issue that I observed is that the IIS fields don't appear to be in the index template populated when Filebeat starts up. Have I missed something in the setup/build of this new module? For configuring the Kibana dashboards I manually edited the index template.

If you run make update at the root of the project it will take care of the templates and the docs, your will have to add the modified file on this PR. The missing files are making this PR fails CI.

Concerning moving forward, I think @ruflin and myself agree that we can take this PR and give part of attribution in #5685

We have some responsability to take, we have been a bit too slow to review it.

ph · 2018-01-23T14:02:08Z

filebeat/docs/modules/iis.asciidoc

+
+The IIS module was tested with logs from version 10.
+
+Note: Running IIS on anything other than Windows is not supported.


Do we really need that mention? We could always still parse IIS formatted log on a Linux machine.

Agreed. I've removed this comment.

ph · 2018-01-23T14:14:22Z

filebeat/module/iis/access/_meta/fields.yml

+      type: long
+      description: >
+        The HTTP response code.
+    - name: sub_status


I am looking at the https://msdn.microsoft.com/en-us/library/windows/desktop/aa814385(v=vs.85).aspx, I understand all the snake case reference now. :)

ph · 2018-01-23T14:20:24Z

filebeat/module/iis/access/_meta/fields.yml

+      type: long
+      description: >
+        The Windows status code.
+    - name: time_taken_ms


resquest_time_ms maybe?

Agreed, I've changed this to request_time_ms.

ph · 2018-01-23T14:23:53Z

filebeat/module/iis/access/test/test.log

+#Version: 1.0
+#Date: 2018-01-01 10:11:12
+#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
+2018-01-01 10:11:12 W3SVC1 MACHINE-NAME 127.0.0.1 GET / - 80 - 85.181.35.98 HTTP/1.1 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:57.0)+Gecko/20100101+Firefox/57.0 - - example.com 200 0 0 123 456 789


Maybe one record could have a query string?

Agreed, I've added a query string example to the first log entry in this file. The query string is quite simple (I've just lifted this from the Nginx module test file). Let me know if you want a fancier query string here.

Thanks for your feedback @ph. All review items should be addressed now. Let me know if there is anything else you'd like me to pick up.

ph · 2018-01-23T14:41:36Z

CHANGELOG.asciidoc

@@ -32,6 +32,8 @@ https://github.com/elastic/beats/compare/v6.0.0-beta2...master[Check the HEAD di
 *Filebeat*
 - Switch to docker prospector in sample manifests for Kubernetes deployment {pull}5963[5963]

+- Add IIS module. {pull}6127[6127]


Add IIS module to parse access log and error log.

BongoEADGC6 · 2018-01-31T16:22:26Z

@cgwrench Thanks for taking this on. I'm willing to help however I can to get this moved into the beats product as soon as possible.

ph · 2018-01-31T16:32:34Z

jenkins test this please

ph · 2018-01-31T16:43:58Z

@BongoEADGC6 I think what we can do is, get this one merged and rebase changes on a new PR?

cc @ruflin

ph

LGTM but WFSG

Waiting on CI to run everything, I've also restarted the travis filebeat job, It appears to fails because of some network error and weirdness of old pip version.

BongoEADGC6 · 2018-01-31T17:02:40Z

@ph No problem. I merged @cgwrench repo into mine and updated my pull request #5685 if you want to do that as well.

ruflin · 2018-02-03T04:50:20Z

I think there are some fixes needed in the tests: https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-linux/3041/beat=filebeat,label=ubuntu/testReport/junit/test_modules/Test/test_modules/

@ph I added you also as reviewer on to move this forward: #5685

ph · 2018-02-05T20:25:49Z

@ruflin 👍 I will take a look at #5685, thanks @cgwrench for updating this PR :)

cgwrench · 2018-02-05T20:34:32Z

@ph No problem. @ruflin Thanks for pointing this out that failing test, I think I've addressed this now. Let me know if there is anything else I can pick up.

ph · 2018-02-05T20:53:48Z

@cgwrench I've restarted the test, there seems to be some network/artifact issues unrelated to your changes, I am keeping an eye open.

ph · 2018-02-08T14:18:32Z

jenkins test this please

ph · 2018-02-12T16:48:30Z

@cgwrench We have one error with the test, I believe its because we now send input.type in the document see:

https://github.com/elastic/beats/blob/master/filebeat/module/logstash/slowlog/test/slowlog-plain.log-expected.json#L34-L36

AssertionError: The following expected object was not found:
 {u'error': {u'reason_phrase': u'Hostname', u'server_port': u'80', u'remote_port': u'2780', u'url': u'/ThisIsMyUrl.htm', u'geoip': {u'continent_name': u'Europe', u'country_iso_code': u'DE', u'location': {u'lat': 51, u'lon': 9}}, u'response_code': u'400', u'queue_name': u'-', u'server_ip': u'127.0.0.1', u'http_version': u'1.1', u'method': u'GET', u'remote_ip': u'85.181.35.98'}}
Searched in: 
[{u'iis': {u'error': {u'reason_phrase': u'Hostname', u'server_port': u'80', u'remote_port': u'2780', u'url': u'/ThisIsMyUrl.htm', u'geoip': {u'continent_name': u'Europe', u'city_name': u'Berlin', u'location': {u'lat': 52.5167, u'lon': 13.4}, u'region_name': u'Land Berlin', u'country_iso_code': u'DE'}, u'response_code': u'400', u'queue_name': u'-', u'server_ip': u'127.0.0.1', u'http_version': u'1.1', u'method': u'GET', u'remote_ip': u'85.181.35.98'}}, u'beat': {u'hostname': u'ba2261c759f9', u'name': u'ba2261c759f9', u'version': u'7.0.0-alpha1'}, u'@timestamp': u'2018-01-01T09:10:11.000Z', u'read_timestamp': u'2018-02-08T14:40:04.429Z', u'fileset': {u'name': u'error', u'module': u'iis'}, u'source': u'/go/src/github.com/elastic/beats/filebeat/module/iis/error/test/test.log', u'offset': 384, u'input': {u'type': u'log'}, u'prospector': {u'type': u'log'}}, {u'iis': {u'error': {u'reason_phrase': u'ConnLimit', u'server_port': u'80', u'remote_port': u'2094', u'url': u'/qos/1kbfile.txt', u'http_version': u'1.1', u'response_code': u'503', u'queue_name': u'-', u'server_ip': u'172.31.77.6', u'method': u'GET', u'remote_ip': u'172.31.77.6'}}, u'beat': {u'hostname': u'ba2261c759f9', u'name': u'ba2261c759f9', u'version': u'7.0.0-alpha1'}, u'@timestamp': u'2018-01-01T08:09:10.000Z', u'read_timestamp': u'2018-02-08T14:40:04.429Z', u'fileset': {u'name': u'error', u'module': u'iis'}, u'source': u'/go/src/github.com/elastic/beats/filebeat/module/iis/error/test/test.log', u'offset': 286, u'input': {u'type': u'log'}, u'prospector': {u'type': u'log'}}, {u'iis': {u'error': {u'reason_phrase': u'Version_N/S', u'server_port': u'80', u'remote_port': u'2894', u'url': u'/', u'geoip': {u'continent_name': u'Europe', u'city_name': u'Berlin', u'location': {u'lat': 52.5167, u'lon': 13.4}, u'region_name': u'Land Berlin', u'country_iso_code': u'DE'}, u'response_code': u'505', u'queue_name': u'-', u'server_ip': u'127.0.0.1', u'http_version': u'2.0', u'method': u'GET', u'remote_ip': u'85.181.35.98'}}, u'beat': {u'hostname': u'ba2261c759f9', u'name': u'ba2261c759f9', u'version': u'7.0.0-alpha1'}, u'@timestamp': u'2018-01-01T10:11:12.000Z', u'read_timestamp': u'2018-02-08T14:40:04.429Z', u'fileset': {u'name': u'error', u'module': u'iis'}, u'source': u'/go/src/github.com/elastic/beats/filebeat/module/iis/error/test/test.log', u'offset': 470, u'input': {u'type': u'log'}, u'prospector': {u'type': u'log'}}, {u'iis': {u'error': {u'reason_phrase': u'Timer_MinBytesPerSecond', u'server_ip': u'127.0.0.1', u'remote_port': u'64388', u'geoip': {u'continent_name': u'Europe', u'city_name': u'Berlin', u'location': {u'lat': 52.5167, u'lon': 13.4}, u'region_name': u'Land Berlin', u'country_iso_code': u'DE'}, u'queue_name': u'-', u'server_port': u'80', u'remote_ip': u'85.181.35.98'}}, u'beat': {u'hostname': u'ba2261c759f9', u'name': u'ba2261c759f9', u'version': u'7.0.0-alpha1'}, u'@timestamp': u'2018-01-01T11:12:13.000Z', u'read_timestamp': u'2018-02-08T14:40:04.429Z', u'fileset': {u'name': u'error', u'module': u'iis'}, u'source': u'/go/src/github.com/elastic/beats/filebeat/module/iis/error/test/test.log', u'offset': 558, u'input': {u'type': u'log'}, u'prospector': {u'type': u'log'}}]

cgwrench · 2018-02-12T19:34:20Z

@ph Thanks for pointing this out. I couldn't see input.type in the array of documents searched. I think the failing test is because the expected geoip object isn't correct. I fixed this previously for the access fileset and obviously missed this same issue in the error fileset. Hopefully with this latest change all the test now pass. Please let me know if there's something else I've missed.

ph · 2018-02-12T20:46:31Z

jenkins test this please

ph · 2018-02-13T18:03:57Z

@cgwrench 👍 🤗 ❤️ Thanks for the all works :)

cgwrench · 2018-02-13T21:24:13Z

@ph No problem, thanks for all your input!

cgwrench and others added 10 commits January 20, 2018 22:36

Add IIS Filebeat module

31192e1

Correct IIS version number in IIS module docs

8837a5c

Add Filebeat IIS module error file set

1cb5f2c

Adjust Filebest IIS access grok patterns

14b8175

Add Filebeat IIS error fileset to config files

1e014cc

Correct names of Filebeat IIS module config files

e2f4846

Add Filebeat IIS module test logs files

f523532

Add Filebeat IIS module Kibana dashboard

307eca8

Update Filebeat IIS module documentation

c48aa01

Add pull request to IIS Filebeat changelog entry

850dbb0

cgwrench mentioned this pull request Jan 21, 2018

[FileBeat] Adding IIS Module for Windows #5685

Closed

ruflin added module review Filebeat Filebeat labels Jan 21, 2018

ph suggested changes Jan 23, 2018

View reviewed changes

ph reviewed Jan 23, 2018

View reviewed changes

cgwrench added 2 commits January 23, 2018 20:53

Update Filebeat IIS module documentation

d1a7243

Add query string sample to IIS Filebeat module tests

37ce52f

ph approved these changes Jan 31, 2018

View reviewed changes

Fix Filebeat IIS module expected test file

e32274a

Fix Filebeat IIS module expected error test file

375df6a

ph merged commit 652b163 into elastic:master Feb 13, 2018

cgwrench deleted the iis-filebeat-module branch February 13, 2018 21:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add IIS Filebeat module #6127

Add IIS Filebeat module #6127

cgwrench commented Jan 21, 2018

karmi commented Jan 21, 2018

elasticmachine commented Jan 21, 2018

cgwrench commented Jan 21, 2018

ruflin commented Jan 21, 2018

cgwrench commented Jan 22, 2018

ph left a comment

ph Jan 23, 2018

cgwrench Jan 23, 2018

ph Jan 23, 2018

ph Jan 23, 2018

cgwrench Jan 23, 2018

ph Jan 23, 2018

cgwrench Jan 23, 2018 •

edited

Loading

cgwrench Jan 23, 2018

ph Jan 23, 2018

BongoEADGC6 commented Jan 31, 2018

ph commented Jan 31, 2018

ph commented Jan 31, 2018

ph left a comment

BongoEADGC6 commented Jan 31, 2018 •

edited

Loading

ruflin commented Feb 3, 2018

ph commented Feb 5, 2018

cgwrench commented Feb 5, 2018

ph commented Feb 5, 2018

ph commented Feb 8, 2018

ph commented Feb 12, 2018

cgwrench commented Feb 12, 2018

ph commented Feb 12, 2018

ph commented Feb 13, 2018

cgwrench commented Feb 13, 2018


		The IIS module was tested with logs from version 10.

		Note: Running IIS on anything other than Windows is not supported.

Add IIS Filebeat module #6127

Add IIS Filebeat module #6127

Conversation

cgwrench commented Jan 21, 2018

karmi commented Jan 21, 2018

elasticmachine commented Jan 21, 2018

cgwrench commented Jan 21, 2018

ruflin commented Jan 21, 2018

cgwrench commented Jan 22, 2018

ph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwrench Jan 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BongoEADGC6 commented Jan 31, 2018

ph commented Jan 31, 2018

ph commented Jan 31, 2018

ph left a comment

Choose a reason for hiding this comment

BongoEADGC6 commented Jan 31, 2018 • edited Loading

ruflin commented Feb 3, 2018

ph commented Feb 5, 2018

cgwrench commented Feb 5, 2018

ph commented Feb 5, 2018

ph commented Feb 8, 2018

ph commented Feb 12, 2018

cgwrench commented Feb 12, 2018

ph commented Feb 12, 2018

ph commented Feb 13, 2018

cgwrench commented Feb 13, 2018

cgwrench Jan 23, 2018 •

edited

Loading

BongoEADGC6 commented Jan 31, 2018 •

edited

Loading