rsyslog request.log.gz problem

[mafiu]

Dear @GUI,

Some day I had a problem with my instance of ES, so I looked at the backup of the request log sended to ES in /opt/api-umbrella/var/log/rsyslog/requests.log.gz

But this file is unreadeable... or I don't know how to read it...

[api-02 rsyslog]$ file requests.log.gz
requests.log.gz: data
[api-02 rsyslog]$ gzip -d requests.log.gz
gzip: requests.log.gz: not in gzip format

I'm using API-Umbrella v0.13

Thanks

[GUI]

I've unfortunately seen the same problem on our servers, which makes this requests.log.gz file fairly useless in the current v0.13 release. The intention of this extra log file was exactly for this purpose--to try and make restoring data into Elasticsearch easier if something goes wrong with Elasticsearch, but unfortunately the file seems corrupted somehow.

I haven't had a chance to figure where the issue lies, but I suspect it's somehow related to rsyslog and log rotation (the file starts out as valid gzip, but after log rotation kicks in, it seems to become corrupted). But I'm planning to look into this to try and make sure we can resolve it before the next v0.14 release.

In the mean time, it should be possible to restore some of the missing data based on the nginx access logs (which shouldn't have this corruption problem). The nginx logs won't have all the normal detail (it's missing various timing information, and request/response headers), but it should give you the basic request analytics. However, restoring from the nginx logs will require some custom scripts to parse the nginx access log format and load it into elasticsearch. I have some scripts somewhere that I've used to do this previously that I'm happy to dig up and share. I'm actually needing to do a similar restore task for one of our API Umbrella instances, so hopefully I'll be tackling this sometime in the next week and I can share the scripts when I get to that task.

[mafiu]

Thanks @GUI,

I'm gonna take a look too, and if I found the solution I'm gonna push it.

[GUI]

@mafiu: I've added some scripts to restore missing analytics based on the nginx logs. Details on how to use the scripts (and caveats) are here: https://github.com/NREL/api-umbrella/tree/master/scripts/import_access_logs Hope that helps!

More generally, I still do want to make sure we get the corrupt gzip files from rsyslog fixed for the next API Umbrella release. Having those log files would make restoring the analytics much easier and more complete in the event things go awry. I think the gzip corruption issue is a bug with rsyslog and log rotation, since some other people seem to be reporting similar issues with rsyslog here: rsyslog/rsyslog#933

[GUI]

These corrupt files request.log.gz files should be fixed in v0.14.0.