[Filebeat] filebeat-7.15.0-apache-error-pipeline doesn't recognize apache trace log levels

[renzedj]

Apache error module ingest pipeline (filebeat-7.15.0-apache-error-pipeline) doesn't recognize apache trace1 - trace8 log levels.

The following entry in the Apache error log:

[Wed Oct 20 19:20:59.121211 2021] [rewrite:trace3] [pid 121591:tid 140413273032448] mod_rewrite.c(470): [client 10.121.192.8:38350] 10.121.192.8 - - [api-dev.onintranet.com/sid#55a374e851c8][rid#7fb438083ac0/initial] applying pattern '^/bankimport/?(.*)$' to uri '/'

will generate the following error:

Provided Grok expressions do not match field value: [[Wed Oct 20 19:20:59.121211 2021] [rewrite:trace3] [pid 121591:tid 140413273032448] mod_rewrite.c(470): [client 10.121.192.8:38350] 10.121.192.8 - - [api-dev.onintranet.com/sid#55a374e851c8][rid#7fb438083ac0/initial] applying pattern '^/bankimport/?(.*)$' to uri '/']

Resolution is to update the pipeline to include an APACHE_LOG_LEVEL definition. Working pipeline is as follows:

{
  "filebeat-7.15.0-apache-error-pipeline" : {
    "description" : "Pipeline for parsing apache error logs",
    "processors" : [
      {
        "set" : {
          "value" : "{{_ingest.timestamp}}",
          "field" : "event.ingested"
        }
      },
      {
        "rename" : {
          "field" : "message",
          "target_field" : "event.original"
        }
      },
      {
        "grok" : {
          "field" : "event.original",
          "patterns" : [
            """\[%{APACHE_TIME:apache.error.timestamp}\] \[%{APACHE_LOG_LEVEL:log.level}\]( \[client %{IPORHOST:source.address}(:%{POSINT:source.port})?\])? %{GREEDYDATA:message}""",
            """\[%{APACHE_TIME:apache.error.timestamp}\] \[%{DATA:apache.error.module}:%{APACHE_LOG_LEVEL:log.level}\] \[pid %{NUMBER:process.pid:long}(:tid %{NUMBER:process.thread.id:long})?\]( \[client %{IPORHOST:source.address}(:%{POSINT:source.port})?\])? %{GREEDYDATA:message}"""
          ],
          "pattern_definitions" : {
            "APACHE_TIME" : "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}",
            "APACHE_LOG_LEVEL" : "(emerg|alert|crit|error|warn|notice|info|debug|trace[1-8])"
          },
          "ignore_missing" : true
        }
      },
      {
        "grok" : {
          "ignore_failure" : true,
          "field" : "message",
          "patterns" : [
            "File does not exist: %{URIPATH:file.path}, referer: %{URI:http.request.referrer}",
            "File does not exist: %{URIPATH:file.path}"
          ],
          "ignore_missing" : true
        }
      },
      {
        "date" : {
          "formats" : [
            "EEE MMM dd H:m:s yyyy",
            "EEE MMM dd H:m:s.SSSSSS yyyy"
          ],
          "if" : "ctx.event.timezone == null",
          "field" : "apache.error.timestamp",
          "target_field" : "@timestamp",
          "on_failure" : [
            {
              "append" : {
                "field" : "error.message",
                "value" : "{{ _ingest.on_failure_message }}"
              }
            }
          ]
        }
      },
      {
        "date" : {
          "field" : "apache.error.timestamp",
          "target_field" : "@timestamp",
          "formats" : [
            "EEE MMM dd H:m:s yyyy",
            "EEE MMM dd H:m:s.SSSSSS yyyy"
          ],
          "timezone" : "{{ event.timezone }}",
          "if" : "ctx.event.timezone != null",
          "on_failure" : [
            {
              "append" : {
                "field" : "error.message",
                "value" : "{{ _ingest.on_failure_message }}"
              }
            }
          ]
        }
      },
      {
        "remove" : {
          "ignore_failure" : true,
          "field" : [
            "apache.error.timestamp",
            "_tmp.url_orig"
          ]
        }
      },
      {
        "set" : {
          "field" : "event.kind",
          "value" : "event"
        }
      },
      {
        "set" : {
          "field" : "event.category",
          "value" : "web"
        }
      },
      {
        "script" : {
          "if" : "ctx?.log?.level != null",
          "lang" : "painless",
          "source" : """def err_levels = ["emerg", "alert", "crit", "error", "warn"]; if (err_levels.contains(ctx.log.level)) {
  ctx.event.type = "error";
} else {
  ctx.event.type = "info";
}"""
        }
      },
      {
        "grok" : {
          "ignore_missing" : true,
          "patterns" : [
            "^(%{IP:source.ip}|%{HOSTNAME:source.domain})$"
          ],
          "field" : "source.address"
        }
      },
      {
        "geoip" : {
          "field" : "source.ip",
          "target_field" : "source.geo",
          "ignore_missing" : true
        }
      },
      {
        "geoip" : {
          "target_field" : "source.as",
          "properties" : [
            "asn",
            "organization_name"
          ],
          "ignore_missing" : true,
          "database_file" : "GeoLite2-ASN.mmdb",
          "field" : "source.ip"
        }
      },
      {
        "rename" : {
          "field" : "source.as.asn",
          "target_field" : "source.as.number",
          "ignore_missing" : true
        }
      },
      {
        "rename" : {
          "target_field" : "source.as.organization.name",
          "ignore_missing" : true,
          "field" : "source.as.organization_name"
        }
      },
      {
        "script" : {
          "description" : "This script processor iterates over the whole document to remove fields with null values.",
          "source" : """void handleMap(Map map) {
  for (def x : map.values()) {
    if (x instanceof Map) {
        handleMap(x);
    } else if (x instanceof List) {
        handleList(x);
    }
  }
  map.values().removeIf(v -> v == null);
}
void handleList(List list) {
  for (def x : list) {
      if (x instanceof Map) {
          handleMap(x);
      } else if (x instanceof List) {
          handleList(x);
      }
  }
}
handleMap(ctx);
""",
          "lang" : "painless"
        }
      }
    ],
    "on_failure" : [
      {
        "set" : {
          "field" : "error.message",
          "value" : "{{ _ingest.on_failure_message }}"
        }
      }
    ]
  }
}

Have not validated whether this is also required for the access pipeline or not.

For confirmed bugs, please report:

• Version: 7.15.0
• Operating System: Kubernetes deployment
• Steps to Reproduce: Generate error in apache error log on client machine with trace1 - trace8 log level.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[jsoriano]

Hey @drenze-athene, thanks for reporting! I have opened #28717 and elastic/integrations#2064 to fix this in Beats and in the Apache integration.

[renzedj]

@jsoriano - NP. Ended up being a fairly easy fix in our instance. Didn't know what else in the pipeline might need to be updated, though.

[jsoriano]

Ended up being a fairly easy fix in our instance. Didn't know what else in the pipeline might need to be updated, though.

@drenze-athene what did you change in your instance? Current pipeline cannot parse this log line containing trace3.

[renzedj]

@jsoriano -

I have a CI/CD pipeline that uploads templates and ingest pipelines in our environment, and we don't override them once they've been uploaded (i.e., we don't deploy our beats with setup enabled). So after it was deployed, I edited the pipeline to include the following pattern definition:

"APACHE_LOG_LEVEL" : "(emerg|alert|crit|error|warn|notice|info|debug|trace[1-8])"

I included the complete functioning pipeline in my OP.

[jsoriano]

Oh ok, so the change in the pipeline would be needed. Thanks!

[renzedj]

Oh ok, so the change in the pipeline would be needed. Thanks!

Absolutely. I looked at your change, didn't think of doing it that way. I just took the list of log levels specified in the Apache documentation and turned it into a regex. Yours was much easier.