Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.ops: add ecamp3-logging #5039

Merged
merged 1 commit into from
May 19, 2024
Merged

.ops: add ecamp3-logging #5039

merged 1 commit into from
May 19, 2024

Conversation

BacLuc
Copy link
Contributor

@BacLuc BacLuc commented Apr 23, 2024

Finally i have a branch again which works.
Now with a lot less yaml.

This adds the EFFK (Elasticsearch, fluentbit, fluentd, Kibana) stack to the ops folder.
Kibana and Elasticsearch taken from
https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes

For the full story, see: #4775

@usu
Copy link
Member

usu commented Apr 27, 2024

Nice, thank you @BacLuc

I was not that deeply involved in the discussion in #4775. So this is now a mix between option 1 and 2?
Do you have an estimation, how much additional nodes we'd need to run it?
Is this deployed to dev already to test out?

@BacLuc
Copy link
Contributor Author

BacLuc commented Apr 27, 2024

I was not that deeply involved in the discussion in #4775.
So this is now a mix between option 1 and 2?

Yes, but with the fluent operator and fluentbit and fluentd instead of filebeat.
You see the list of the tried options here: #4775 (comment)
This is the last one

Do you have an estimation, how much additional nodes we'd need to run it?

Currently on dev it looks like this:

NAME CPU(cores) MEMORY(bytes)
elasticsearch-cluster-0 18m 977Mi
fluent-bit-4mh29 4m 56Mi
fluent-bit-b92zd 3m 52Mi
fluent-bit-fdmzw 3m 47Mi
fluent-bit-s5rgz 4m 48Mi
fluent-operator-7c5d78987c-tbx6l 1m 49Mi
fluentd-0 6m 155Mi
kibana-7cc44b6d66-jllsq 86m 523Mi
Total 1907Mi

So we would need another node. Maybe 2 if elasticsearch contains a lot of logs, we will see.

Is this deployed to dev already to test out?

Yes, you can do a port forward with
k port-forward services/kibana 5601:5601

BacLuc added a commit to BacLuc/ecamp3 that referenced this pull request Apr 27, 2024
That we don't need kubectl to access these internal services.
Can be used for the kibana from ecamp#5039,
or the kubernetes dashboard from ecamp#4134
.ops/ecamp3-logging/README.md Show resolved Hide resolved
.ops/ecamp3-logging/files/kibana/kibana-objects.ndjson Outdated Show resolved Hide resolved
@usu usu added the Meeting Discuss Am nächsten Core-Meeting besprechen label May 15, 2024
Copy link
Member

@carlobeltrame carlobeltrame left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks a lot!

This adds the EFFK (Elasticsearch, fluentbit, fluentd, Kibana) stack
to the ops folder.
Kibana and Elasticsearch taken from
https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes
carlobeltrame added a commit to ecamp/ecamp-site that referenced this pull request May 15, 2024
@BacLuc BacLuc removed the Meeting Discuss Am nächsten Core-Meeting besprechen label May 15, 2024
@BacLuc BacLuc added this pull request to the merge queue May 19, 2024
Merged via the queue into ecamp:devel with commit c88c5ba May 19, 2024
36 of 37 checks passed
@BacLuc BacLuc deleted the add-ecamp3-logging-4 branch May 26, 2024 11:41
@BacLuc BacLuc mentioned this pull request Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants