Update elk docker image and swap out logstash

[SwooshyCueb]

Updating this docker image as part of the effort to figure out what's going on with our audit plugin training setup.

Still a work-in-progress. Should probably be using alpine or something instead of relying on systemd.
Using elasticsearch 7 for now, as the kibana in the repo for elasticsearch 8 does not support importing json saved objects.

[trel]

oh interesting - how would one not have to build a dashboard by hand... must be some way to import a thing...

but yes, any progress is good. we'll get it built and tried soon enough.

[SwooshyCueb]

Yeah, they have their own ndjson format we can export with 7 and import into 8. Haven't gotten that far yet though.

[trel]

Ah, very good. We do love a nice text format.

[SwooshyCueb]

Updated container to run the still-relevant bits of the old startup script and create the index pattern, visualization objects, and dashboard automatically on first run.
Still on elk 7 for now. Now that we have a good saved object payload for 7, updating to 8 shouldn't take very long.
I would still like to move away from using systemd in the container at some point, but that's lower priority than just getting everything updated and fixing the amqp header issue.

[SwooshyCueb]

Updated container to elk 8

[trel]

awesome - will definitely need a walkthrough/demo of this.

[SwooshyCueb]

Swapped logstash for a python script.
Works perfectly with 4.2.11 now. Haven't tested with 4.3 yet as I'm having trouble getting it spun up in a container.

[SwooshyCueb]

For 4.3.0, four of the visualizations in the dashboard were empty, so I pointed them at a different rule name.
audit_pep_auth_agent_auth_request_pre to audit_pep_api_authenticate_pre

I think this will make the container incompatible with 4.2, but now it works with 4.3.0.
I'm not going to squash this commit, as it might be useful to have the 4.2-compatible ndjson in the git history.

[korydraughn]

So the only difference between 4.2 and 4.3 is the PEP names.

We can make the ndjson file mountable (-v) and require the admin to use the correct file based on the version of iRODS. We'd include ndjson files for 4.2 and 4.3 that live beside the docker file. For example, the irods_audit_elk_stack directory could contain the following:

• Dockerfile
• example_kibana_dashboard_v4.2.ndjson
• example_kibana_dashboard_v4.3.ndjson

Doing that allows us to build a single docker image that supports 4.2 and 4.3. docker-compose would also be supported.

The training slides would require trivial tweaks to sync with this scheme.

[trel]

this is quite promising.

[SwooshyCueb]

I'm not a fan of that particular solution. I think there's a way we can set up the visualizations to look for either pep name, that way we can have a single ndjson for both 4.2 and 4.3 with no volume mounting needed

[korydraughn]

Are you saying you think there's native support for conditionally doing this in kibana/ndjson/etc?

If yes, I agree that would likely be the better approach, especially if it is simple to grasp (i.e. fewer moving parts). Anything more complicated than a volume mount is probably the wrong way to go.

Eventually, the things related to 4.2 will be dropped leaving only 4.3 (hopefully sooner than later). Keeping that in mind while we discuss the pros/cons of each solution for this will help us make the correct decision.

[SwooshyCueb]

Are you saying you think there's native support for conditionally doing this in kibana/ndjson/etc?

Yes, and I've just pushed up the new ndjson. This container should now work fine with 4.2 and 4.3.0.

[korydraughn]

Nice! Will take a look.

Was there anything you noticed that we need to keep an eye out for in the future?

[korydraughn]

We also need to document (for our future selves) how to tweak the ndjson file.

You could include a README in the irods_audit_elk_stack directory that captures those steps. It could also reference the kibana/elastic docs for whatever version you used to make this all work.

[SwooshyCueb]

Was there anything you noticed that we need to keep an eye out for in the future?

Not so much stuff we need to look out for, but stuff we need to do.

• I'd like to get away from using systemd in the container (stop using systemd in elk stack container #28). It would be nice if we could use alpine as well, but I think it is probably easier maintenance-wise to stick with ubuntu.
• We should remove the start/stop tokens in the audit plugin (Remove __BEGIN_JSON__/__END_JSON__ tokens irods_rule_engine_plugin_audit_amqp#102).
• I think we might need to take a second look at how the json emitted by the audit plugin is constructed (Emit json with proper types. irods_rule_engine_plugin_audit_amqp#103). In the python bridge, we have to convert things from strings to ints and whatnot, and to me it seems like they should already be the proper data-type coming out of the audit plugin ("should" as in "well behaved", not as in "the code looks like it should do this").
• We're using a whole jdk where I think a jre would suffice (Investigate swapping JDK for JRE in elk stack container #29). We can't use Adoptium's jre-only docker images since they aren't set up in a way that the ES packages can use them. They have a couple of blog posts about how we might accomplish this though:
  • https://blog.adoptium.net/2021/10/jlink-to-produce-own-runtime/
  • https://blog.adoptium.net/2021/08/using-jlink-in-dockerfiles/

We also need to document (for our future selves) how to tweak the ndjson file.

Yes, and how to update it for newer versions of kibana.

You could include a README in the irods_audit_elk_stack directory that captures those steps. It could also reference the kibana/elastic docs for whatever version you used to make this all work.

Roger dodger.

[SwooshyCueb]

Considering how long it takes me to write good documentation for this kind of thing, I decided it would be faster to go ahead and rip systemd out of the container instead of documenting the jank that comes from running systemd in a container for our use case. It ended up taking way longer than I anticipated to update the Kibana init.d script, but I still think this was the correct decision.

I've pushed this up now, along with some small container optimizations and a WIP of the README I'm working on.

Getting away from systemd allows us to perform in the Dockerfile all the first-time setup previously handled by firstrun.sh.
Additionally, the container entrypoint now takes an optional --es-java-heap-size argument to set the Elasticsearch Java heap size, with a default of 512m.

[korydraughn]

Should we start reviewing this or wait until the PR status changes from "draft"?

[SwooshyCueb]

Review away! Everything is done except the README

[korydraughn]

Very nice!

The README made me delete some of my comments, so good stuff.

[SwooshyCueb]

A humorous (to me) observation:
While it is not uncommon for my PRs to contain multiple typos prior to review, these typos are usually my own doing.
In this PR, most of the typos are from the vendor-provided files on which the init.d scripts and configuration files are based.

[korydraughn]

Ship it!

[SwooshyCueb]

Who needs to be poked to get this merged?

[korydraughn]

@alanking ping!

[alanking]

Now watching this repo :)