Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

close DO incident #330

Merged
merged 1 commit into from
Jun 22, 2023
Merged

close DO incident #330

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions content/issues/2023-06-21-digitalocean-incident.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,23 @@
---
title: DigitalOcean outage
date: 2023-06-21T11:45:00-00:00
resolved: false
resolvedWhen: 2023-06-21T11:45:00-00:00
resolved: true
resolvedWhen: 2023-06-22T16:45:00-00:00
# Possible severity levels: down, disrupted, notice
severity: disrupted
affected:
- ci.jenkins.io
section: issue
---

[Final message]
Incident is reported officially by Digital Ocean and is finished: <https://status.digitalocean.com/incidents/bclmy32d12p0>.

We've restored all the nominal configuration after health-checking the clusters.

[Initial message]
DigitalOcean infrastructure is having a network outage.
We see a high rate of TCP errors on both Kubernetes clusters `doks` (agents for ci.jenkins.io) and `doks-public` (Artifact Caching Proxy for these agents).
We are disabling temporarily agent scheduling in DigitalOcean: all ci.jenkins.io containers workload will run in AWS or Azure instead.

More details in <https://github.com/jenkins-infra/helpdesk/issues/3633>.