Liveness/Readiness probes

[Mrkazik99]

Description:

Readiness probe: Will check all dependencies of each component (like S3, DB and other). It should be as lightweight as possible

Liveness probe: This should be independent from 3rd party services outage, connection errors might be considered (if Database is not accepting connections this probe should not be affected, but if pod cannot resolve hostname, then restart of component might be helpful)

TL;DR, liveness probe is causing restart of application, but readiness will just disconnect it from service/in case of correct PDB + HA setup will instantly schedule another replica to handle traffic.

We also need some additional endpoint that will detail readiness probe (or it can be merged if it won't affect performance of probe) that will tell about status of 3rd party connections, like in case of conversion it will inform us about issue with S3 connection or some redis issues.

---

Use Case:

K8S deployment of application mainly but not only

---

Benefits:

1. Easier debug process in k8s and with this "detailed probe" docker might be easier as well
2. We will be k8s native
3. First steps into monitoring implementation, this "detailed probe" will be good endpoint to scrape metrics using prometheus

---

Possible Implementation (if applicable):

--

---

Additional Information:

--

---

[dsonck92]

Proposed implementation:

• Keep a repository of statuses inside the app, defaulting to "not ready", globally accessible
• Let various components post updates they know about to the repository
• Repository implements the ready handler

Having a repository that gets updates posted from other systems allow the probe to be very efficient, albeit slightly lagging behind the actual situation. If we make the repository a map/struct of errors, then applications can post their failures for feedback utilizing the error framework from go. A nil would be considered "good to go"