Increase support of log formats in haproxy filebeat module

[jsoriano]

During the investigation of elastic/beats#8301 issue we identified some patterns that could be added to the initial module implemented for haproxy (#8014):

• Rename fileset from http to log, as the same log file can contain info about tcp connections and other general errors too Haproxy fileset http renamed to log beats#8405
• Add supported formats to documentation
• Check if http log entries without captured headers can be parsed, if not, make captured headers optional. Add tests for that in any case. Support haproxy logs without captured headers beats#9463 Support haproxy log lines without captured headers beats#9958
• Support logs generated with option tcplog. Haproxy filebeat TCP log format beats#8526 Haproxy filebeat module tcp and default formats beats#8637

Feb  6 12:12:56 localhost haproxy[14387]: 10.0.1.2:33313 [06/Feb/2009:12:12:51.443] fnt bck/srv1 0/0/5007 212 -- 0/0/0/0/3 0/0

• Support the default (though also deprecated) format used when no other option is used. Filebeat HAproxy Default log format added beats#8428 Haproxy filebeat module tcp and default formats beats#8637

Feb  6 12:12:09 localhost haproxy[14385]: Connect from 10.0.1.2:33312 to 10.0.3.31:8012  (www/HTTP)

• Add pattern for server UP/DOWN logs like Extend filebeat logstash patterns for the haproxy modul beats#21332

Sep 13 15:51:16 debian8-haproxy haproxy[5988]: Server mysvc/myserver01 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

• Add pattern for TLS version and cipher fields Filebeat: Add HAProxy module support for TLS version and cipher log fields beats#29318
• Add pattern for logs of HTTP/2.0 requests (see https://discuss.elastic.co/t/haproxy-log-pipeline-fails-to-extract-http-request-method-for-http-2-0-requests/308956)
• Add catch-all pattern for general log messages like:

Sep 13 15:51:15 debian8-haproxy haproxy[5988]: backend myservers has no server available!

• Investigate issue with syslog log lines Filebeat haproxy module cannot parse syslog line beats#13995
• Add protocol to TCP and HTTP log lines in haproxy.mode. Right now, only HAProxy default format outputs a mode field pointing if the log line is either HTTP or TCP. But the HTTP and TCP logs doesn't actually show anything like this.
• Parse timestamps taking into account the timezone for logs without timezone.
• Parse (haproxy version is 2.2)

Apr 28 16:09:58 ha1.prod.ad.qqqcore.com haproxy[18923]: 119.169.133.47:50040 [28/Apr/2022:16:09:58.167] Advertstream_Log~ Advertstream_Log/log2.prod.ad.qqqcore.com 0/0/2/32/+34 200 +313 - - --VN 116/106/1/1/0 0/0 {|l.qqqcore.com||https://qwersimon.com/} {Apache|57|max-age=||} \\\"GET https://l.qqqcore.com/a/log/view/?c=3vUCAFGazq16o_BmABdcR3-BMkXpE6O-3i1M7PyulK3onD3Z1cjvbl-qUdo_wrcYlXJUe1kU-CD48n-9QWED-lfd2vXLBzp6xQiMOoBSfYfo6Bk9qMGPn901IK2Cs0SHewmpxeNKa7Y4AYMiq9dAb-hSHEsku-ijbNiDmPwh5bAp-NR22OdD6ZlJ-7g0rGPF_mtfW3XWaFuUHLqDeu6mIyMHvbf95aPl0AZt481_2b_ujFh2eTEvK0q_dvjfhWr4P_w1_M24LKm_ipHcmzwmXVjdWzMQGxPFeLVA9YuB1akMuOLwFYneJCVa5foi3WTVyBIvwiMpzbYcSfGl5JVJSNq8VsHh5ZyA9GdqnCBI3V3VcPiBwxQZ0Z1fsCEeo29mj4_WmCPFtEYKUTNJYTTcBaNNUZh_cypX&impid=2327760487204226&&r=&npbk=0&dispatcher=&k=&b=204012&zoneid=232776&siteid=11081&a=ae-d&bidder=goodad&earning=3.4019999999999997&currency=EUR&auctionId=8739afe7-ad6b-4676-b9ea-05cb68871be6&adId=11954862d0bfbe6&creativeId=0&testId=0&domain=&country=XX&device=DESK&auctions=adaccess-0_adaccess-0_adpone-0.

[sayden]

I have just added the last bullet about adding the protocol to the module

[ycombinator]

@jsoriano Should elastic/beats#15488 be added to the list in this issue description?

[jsoriano]

@ycombinator in principle http log should be already supported, so if elastic/beats#15488 can be confirmed this would be a bug.

[arshpunia]

@jsoriano Is there any workaround available for general log messages such as:
Sep 23 16:12:51 <hostname> haproxy[22368]: Proxy My_Web_Server started.

[jsoriano]

@jsoriano Is there any workaround available for general log messages such as:
Sep 23 16:12:51 <hostname> haproxy[22368]: Proxy My_Web_Server started.

Not that I know, we would need to add a "catch-all" grok pattern in the pipeline: https://github.com/elastic/beats/blob/fab73dd058e152ac67918893d9f74bcc6aac1bbf/filebeat/module/haproxy/log/ingest/pipeline.yml

[andrask]

elastic/beats#13995 was closed automatically but it has not been fixed, I believe.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[anmironov]

added new item to the description

Apr 28 16:09:58 ha1.prod.ad.qqqcore.com haproxy[18923]: 119.169.133.47:50040 [28/Apr/2022:16:09:58.167] Advertstream_Log~ Advertstream_Log/log2.prod.ad.qqqcore.com 0/0/2/32/+34 200 +313 - - --VN 116/106/1/1/0 0/0 {|l.qqqcore.com||https://qwersimon.com/} {Apache|57|max-age=||} \\\"GET https://l.qqqcore.com/a/log/view/?c=3vUCAFGazq16o_BmABdcR3-BMkXpE6O-3i1M7PyulK3onD3Z1cjvbl-qUdo_wrcYlXJUe1kU-CD48n-9QWED-lfd2vXLBzp6xQiMOoBSfYfo6Bk9qMGPn901IK2Cs0SHewmpxeNKa7Y4AYMiq9dAb-hSHEsku-ijbNiDmPwh5bAp-NR22OdD6ZlJ-7g0rGPF_mtfW3XWaFuUHLqDeu6mIyMHvbf95aPl0AZt481_2b_ujFh2eTEvK0q_dvjfhWr4P_w1_M24LKm_ipHcmzwmXVjdWzMQGxPFeLVA9YuB1akMuOLwFYneJCVa5foi3WTVyBIvwiMpzbYcSfGl5JVJSNq8VsHh5ZyA9GdqnCBI3V3VcPiBwxQZ0Z1fsCEeo29mj4_WmCPFtEYKUTNJYTTcBaNNUZh_cypX&impid=2327760487204226&&r=&npbk=0&dispatcher=&k=&b=204012&zoneid=232776&siteid=11081&a=ae-d&bidder=goodad&earning=3.4019999999999997&currency=EUR&auctionId=8739afe7-ad6b-4676-b9ea-05cb68871be6&adId=11954862d0bfbe6&creativeId=0&testId=0&domain=&country=XX&device=DESK&auctions=adaccess-0_adaccess-0_adpone-0.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!