Security scanning for SSB/SOLR (test)

[jbrown-xentity]

User Story

In order to meet SI-3, data.gov security wants a scanning tool performing scanning and alerting on malware/any unexpected file system changes.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN Falco (or equivalent) is deployed in EKS and scanning SOLR
  WHEN an unexpected/bad file is added to the SOLR container
  THEN an alert is generated

Background

Related to #3797, would replace the necessity of regular SOLR restarts.

Security Considerations (required)

Related to SI-3.

Sketch

This may change, but current plan is to:

• install as a daemonSet
• helm install from your shell, and testing it out
• If it works, then you just have to add a(nother) helm_release resource to the Terraform in the EKS brokerpak's provision-k8s directory to automate that

[FuhuXia]

Falco installed on a test EKS cluster in staging, Time to config it.

$ helm install falco falcosecurity/falco
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: <...>
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: <...>
NAME: falco
LAST DEPLOYED: Tue May  3 12:50:19 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.

No further action should be required.

Tip:
You can easily forward Falco events to Slack, Kafka, AWS Lambda and more with falcosidekick.
Full list of outputs: https://github.com/falcosecurity/charts/falcosidekick.
You can enable its deployment with `--set falcosidekick.enabled=true` or in your values.yaml.
See: https://github.com/falcosecurity/charts/blob/master/falcosidekick/values.yaml for configuration values.

[hkdctol]

@FuhuXia continuing on it for now.

[FuhuXia]

Events were generated when suspicious commands were run on new containers

$ kubectl run --rm --restart=Never -it --image=busybox -- bash
/ # mkdir /root/aaa
/ # cd ~/aaa
~/aaa # touch asdf
~/aaa #

logs:
16:18:25.363603618: Error File below / or /root opened for writing (user=root user_loginuid=-1 command=touch asdf parent=sh file=/root/aaa/asdf program=touch container_id=c40f893dc3f0 image=busybox) k8s.ns=default k8s.pod=bash container=c40f893dc3f0 k8s.ns=default k8s.pod=bash container=c40f893dc3f0

$ kubectl run --rm --restart=Never -it --image=ubuntu -- bash
root@bash:/# apt-get update

logs:
16:46:50.369247036: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=bash container=c4467cdfd8c6 shell=bash parent= cmdline=bash terminal=34816 container_id=c4467cdfd8c6 image=ubuntu) k8s.ns=default k8s.pod=bash container=c4467cdfd8c6
16:46:56.657919553: Error Package management process launched in container (user=root user_loginuid=-1 command=apt-get update container_id=c4467cdfd8c6 container_name=k8s_bash_bash_default_de5edbeb-79a1-429e-95bf-5fe73740f18b_0 image=ubuntu:latest) k8s.ns=default k8s.pod=bash container=c4467cdfd8c6 k8s.ns=default k8s.pod=bash container=c4467cdfd8c6

[FuhuXia]

Installed falcosidekick and add the slack webhook, alerts were sent to slack channel.

$ helm upgrade -i falco falcosecurity/falco --set falcosidekick.enabled=true --set falcosidekick.config.slack.webhookurl="https://hooks.slack.com/services/SOMECHANNEL/WEBKOOKKEY"

Alert screenshots:

[mogul]

This is SO GREAT. Please make the Slack webhook URL a variable when you add this to the Terraform.

[FuhuXia]

Still need to create custom rule specific for Solr classic/cloud security.

[hkdctol]

Will put on hold for the time being pending ECS work.

[hkdctol]

Moving back to Icebox for now

[nickumia-reisys]

This is not important for data.gov anymore because of Solr on ECS,

• [Tiny EPIC] Support SOLR standalone on ECS #3826

But the scanning mentality would still be useful for other EKS applications. This is a good historical ticket, but I don't think any more work will be done here.