Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change data dir of teleport #3797

Closed
njuettner opened this issue Dec 4, 2024 · 1 comment
Closed

Change data dir of teleport #3797

njuettner opened this issue Dec 4, 2024 · 1 comment
Assignees
Labels
team/shield Team Shield

Comments

@njuettner
Copy link
Member

njuettner commented Dec 4, 2024

We currently store the sessions on /var/lib/teleport, but maybe we should consider to change that
https://github.com/giantswarm/cluster/blob/7382a52e2861cc78548d3e98d4ab204eda696cc5/helm/cluster/files/etc/teleport.yaml#L3

I debuged an issue in alba/seu01 where disk space is low on a node.

Common storage for pods is under /var/lib/kubelet on CAPA but there we also store teleport sessions.

When running out of disk we might have an issue that we cannot login on that nodes anymore due to

Original Error: *trace.AccessDeniedError failed to execute command /var/lib/teleport/log/upload/streaming/default/multi error:  no space left on device

I think we should fix that ASAP

Slack Thread: https://gigantic.slack.com/archives/C02FL4EAADD/p1733322446643659

Solution:

  • Gather insights how much data we store for teleport (/var/lib/teleport)
  • Changing it root directory to isolate this critical component from pod storage abuse, by default we have ~5G for root or we create a separate disk like /var/lib/kubelet if we need more space
@njuettner njuettner added this to Roadmap Dec 4, 2024
@njuettner njuettner converted this from a draft issue Dec 4, 2024
@ssyno ssyno added the team/shield Team Shield label Dec 4, 2024
@ssyno ssyno removed the status in Roadmap Dec 4, 2024
@ssyno ssyno moved this to Inbox 📥 in Roadmap Dec 4, 2024
@ssyno ssyno self-assigned this Dec 4, 2024
@stone-z
Copy link
Contributor

stone-z commented Dec 12, 2024

Merged, waiting for release

@stone-z stone-z closed this as completed Dec 12, 2024
@github-project-automation github-project-automation bot moved this from Inbox 📥 to Done ✅ in Roadmap Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team/shield Team Shield
Projects
Archived in project
Development

No branches or pull requests

3 participants