Metricbeat: Apache 2.4.23 field not found errors

[profilernz]

I'm getting:

metricbeat[6664]: schema.go:36: Error on field 'total': Key ConnsTotal not found
metricbeat[6664]: schema.go:36: Error on field 'closing': Key ConnsAsyncClosing not found
metricbeat[6664]: schema.go:36: Error on field 'writing': Key ConnsAsyncWriting not found
metricbeat[6664]: schema.go:36: Error on field 'keep_alive': Key ConnsAsyncKeepAlive not found

Running Metricbeat 5.0.1 with Apache 2.4.23 (from https://ius.io/) on Centos 7:

httpd -version
Server version: Apache/2.4.23 (CentOS)
Server built: Jul 22 2016 09:11:01

metricbeat -version
metricbeat version 5.0.1 (amd64), libbeat 5.0.1

curl http://127.0.0.1:81/server-status?auto
127.0.0.1
ServerVersion: Apache/2.4.23 (CentOS) OpenSSL/1.0.1e-fips
ServerMPM: prefork
Server Built: Jul 22 2016 09:11:01
CurrentTime: Wednesday, 30-Nov-2016 05:32:43 NZDT
RestartTime: Tuesday, 29-Nov-2016 20:41:30 NZDT
ParentServerConfigGeneration: 1
ParentServerMPMGeneration: 0
ServerUptimeSeconds: 31873
ServerUptime: 8 hours 51 minutes 13 seconds
Load1: 0.00
Load5: 0.01
Load15: 0.05
Total Accesses: 2181
Total kBytes: 53079
CPUUser: 8.4
CPUSystem: 1.22
CPUChildrenUser: 0
CPUChildrenSystem: 0
CPULoad: .0301823
Uptime: 31873
ReqPerSec: .0684278
BytesPerSec: 1705.3
BytesPerReq: 24921.1
BusyWorkers: 1
IdleWorkers: 9
Scoreboard: .W.___....................................................................................................................................................................................................................................................
TLSSessionCacheStatus
CacheType: SHMCB
CacheSharedMemory: 512000
CacheCurrentEntries: 0
CacheSubcaches: 32
CacheIndexesPerSubcaches: 88
CacheIndexUsage: 0%
CacheUsage: 0%
CacheStoreCount: 35
CacheReplaceCount: 0
CacheExpireCount: 35
CacheDiscardCount: 0
CacheRetrieveHitCount: 0
CacheRetrieveMissCount: 89
CacheRemoveHitCount: 0
CacheRemoveMissCount: 0

First reported here:
https://discuss.elastic.co/t/metricbeat-apache-2-4-23-field-not-found-errors/67518

[ruflin]

My first thought was this is related to the Apache version as we test with 2.4.20. But running docker hub image for 2.4.23 I also saw the connection stats, but I'm missing the caching entries you have above.

@profilernz Could you share your apache config?

[radoondas]

@ruflin Some modules in 2.4.X can actually implement their statistics in to server-status. In such case we can easily implement new ones and update the mapping to match.
But I don't think this is the source of the issue.

[ruflin]

@radoondas We recently introduced the concept of raw fields (#3001). We should also add this to apache module so additional fields can be captured. But as you said, in the above case some fields are actually missing. Easy short term solution is to make them optional. But I would like to understand why these don't exist in some of the apache installations.

[radoondas]

@ruflin I have overseen raw fields implementation. Will check it as well.
Those fields doesn't have to exist. Server-status age has 2 modes.

• Default and
• Extended.
  Extended mode have to be enabled in order to get more data.

[profilernz]

ServerRoot "/etc/httpd"

Listen 80

Include conf.modules.d/*.conf

User apache
Group apache

ServerAdmin root@localhost

<Directory />
    AllowOverride none
    Require all denied
</Directory>

DocumentRoot "/var/www/html"

<Directory "/var/www">
    AllowOverride None
    Require all granted
</Directory>

<Directory "/var/www/html">
    Options FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

<Files ".ht*">
    Require all denied
</Files>

ErrorLog "logs/error_log"
LogLevel warn

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
    CustomLog "logs/access_log" combined
</IfModule>

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>

<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>

<IfModule mime_module>
    TypesConfig /etc/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>

AddDefaultCharset UTF-8

<IfModule mime_magic_module>
    MIMEMagicFile conf/magic
</IfModule>

EnableSendfile on

IncludeOptional conf.d/*.conf

Listen 81

<VirtualHost *:81>
    <Location /server-status>
        SetHandler server-status
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1
    </Location>
</VirtualHost>

[ruflin]

@radoondas Based on this the simple current solution is to make Conn optional as it only exists if Extended is enabled?

[profilernz]

Extended seems to be enabled by default since 2.3.6.
I did try turning it on explicitly anyway before reporting this issue, but my stats where exactly the same...

https://httpd.apache.org/docs/2.4/mod/core.html#extendedstatus
Quoting:
"Note that loading mod_status will change the default behavior to ExtendedStatus On, while other third party modules may do the same. Such modules rely on collecting detailed information about the state of all workers. The default is changed by mod_status beginning with version 2.3.6. The previous default was always Off."

[radoondas]

@ruflin I think I might find the issue. It is most likely related to regex used. Let me test it today with the response from this issue.

[ruflin]

@radoondas Thanks. Keep us posted on this one. Feel free to open a PR ;-)

[radoondas]

@ruflin It is not a regex issue.
It is based on the fact, that Fields/Values we are expecting are missing in server-status page. This is normal and we cannot expect all fields as available in the output.

But with our Schema implementation we do expect that they are in the document we apply to.

What options do we have?
1, make all fields not required - can be an issue for other modules.
2, make missing fields only as Warning message - still, this can be misleading, because it is not an issue for Apache.
3, set default values to 0 - I am not sure if this is the best option for all modules.
4, split schema of the object to "must have" and "optional" and later merge them together - we would have to declare which fields are necessary.

[ruflin]

We already have the option to set some fields optional. And this is in this case I think we should do with Con. It is still not 100% clear to me, why the exact same version with same config can have different stats.

Lets make it optional then :)

[radoondas]

Well. 2.4.23 is the latest version (2.4.25 just around the corner). I would like to review changes to server-status page because it seems that there were several changes in past releases based on Changelog.

[ruflin]

@radoondas Cool, do you plan to open a PR?

[radoondas]

@ruflin first, I have to figure out if/what has changed :)

[robertoschwald]

Any news on this? On CentOS 6, the load 1,5,15 fields are not exposed in /server-status, therefore the metricbeat log is filled with errors.

[ruflin]

@robertoschwald No changes yet.
@radoondas Will you have time for this one or should I take it over? Or perhaps @robertoschwald can do a PR as this will make sure it will for sure work with the CentOS 6 version.

[robertoschwald]

Unfortunately, I'm not into the code at all.
On CentOS7, the load 1/5/15 values are also not exposed:

Total Accesses: 2282253
Total kBytes: 1056272
CPULoad: .294215
Uptime: 151641
ReqPerSec: 15.0504
BytesPerSec: 7132.78
BytesPerReq: 473.928
BusyWorkers: 2
IdleWorkers: 16
Scoreboard: _RW_______________..............................................................................................................................................................................................................................................

[ruflin]

@robertoschwald Ok, lets see if @radoondas has some time, otherwise I will take it up. Thanks for posting the raw output, this will help to do some testing.

[Fox32]

My output on Windows looks like @robertoschwald ones, except that "CPULoad" is missing. For some reason this worked in the past 😢

[ruflin]

It's getting more and more interesting. I'm curious why the outputs from Apache are so different ❓

[robertoschwald]

Here is the output from CentOS6:

Total Accesses: 45230
Total kBytes: 86411
CPULoad: .083582
Uptime: 40547
ReqPerSec: 1.1155
BytesPerSec: 2182.28
BytesPerReq: 1956.33
BusyWorkers: 1
IdleWorkers: 74
Scoreboard: _________________________.......................................___________W_____________......................................._________________________.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

[exekias]

Checking apache mod_status.c source some metrics may be missing depending on compilation flags or OS, I'm marking them as optional as suggested by @ruflin

[exekias]

Also I confirmed the issue with CentOS