[docdb] Better log rotation

[ddorian]

Jira Link: DB-2438
Hi

We need a better log rotation policy.
1 way is to keep static naming and allow another process to rotate: google/glog#19 (patch google/glog@58d7f87)

Another way would be to add an option glog_delete_after and a simple bg thread to enforce the policy. For now, having glog_delete_after=<size> should be a good start. It will take us a day or two to implement and probably save us devops trouble in different environments (k8s sidecar comes to mind).

[bmatican]

Opening this up for now, as the bg thread + size flag should not be too complicated.

[iSignal]

We hit this recently in a customer scenario, where glogs took up 20G space over time. We need to be able to have the yb-tserver process automatically cap its logs when run in some of these scenarios where having a separate log prune script is overkill.

[ddorian]

Example: I have a local installation that I've used for a long time. It had 85GB logs in 1 yb-tserver.

[tylarb]

I like the idea of static naming followed by log rotation using an external tool - this allows easy transport of logs to syslog or other OS managed logging systems.

[tylarb]

This feature could be relevant for k8s as well - where we let k8s control the logs size, rather than having a sidecare process / container that only does log rotation.

[kulminaator]

I'm also interested in seeing a more predictable log rotation scheme.

From the top of the hill that i sit on - the hourly or daily rotation scheme that postgres uses makes it very predictable how and where stuff is logged (and also allows easy instructions for external tooling how they can archive - ship - delete log files).

Current scheme where my ERROR log files hardly get any content they will never reach any rotation at all.