Cull CI log groups periodically

[Jimlinz]

Our CI has seed prefixes. This is useful in many ways, especially for troubleshooting, as well as setting up the stage for running pytest tests in parallel. However, one downside is that it creates a handful of new log groups for each seed / CI run (one for each lambda per CI run). This adds up very quickly.

While there is an option to set individual log entry to expire, there isn't an option to delete the log group itself within the geostore stack. AFAIK log group RemovalPolicy.DESTROY isn't an option (see https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_lambda_python_alpha/PythonFunction.html).

This isn't a problem in production, but it is an issue in CI, where CloudWatch logs are pretty much unusable past 10,000 groups.

We should look at various options to keep the number of log groups in check in CI. This would not only keep things tidy, keep costs down (less logs), but also allows us to actually view and search the logs when troubleshooting.

Options:

• GitHub actions to cull logs on a schedule
• Lambda function to cull logs on a schedule
• Find a way to clean up after itself (delete own log groups) during teardown (but would make the logging less useful as they are gone before one could look at it for troubleshooting purposes)
• ???
• Profit

Please feel free to add your thoughts and suggestions below

[l0b0]

We should ideally have tests for what goes into the logs (and we do for a bunch of things), so the logs in the CI instances shouldn't really provide any new info we couldn't gather from the CI output. So I'd be in favour of tearing these down after the tests.

[billgeo]

I agree. And if there was any artifacts we need after CI, should be kept as Github artifacts which can then be configured to be deleted after some set period.

[Jimlinz]

Merged: #2338

[billgeo]

This has been done. And all old log groups will eventually be removed.