Use monitors instead of custom healthcheck.py script to verify service dependencies' health #22951
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KevinMind
changed the title
KevinMind
force-pushed
the
addons-15248-revert
branch
from
d755a98 to
d075bad
Compare
KevinMind
requested review from
a team and
diox
and removed request for
a team
KevinMind
force-pushed
the
addons-15248-fix
branch
4 times, most recently
from
6bb1a40 to
0c00dff
Compare
diox
reviewed
Dec 17, 2024
KevinMind
force-pushed
the
addons-15248-fix
branch
from
0c00dff to
268516a
Compare
KevinMind
force-pushed
the
addons-15248-fix
branch
3 times, most recently
from
f0fe4ec to
825fa3c
Compare
KevinMind
commented
Dec 17, 2024
| f'Expected {settings.DB_CHARSET}, ' | ||
| f'recieved {result[1]}', | ||
| id='setup.E005', | ||
| try: |
There was a problem hiding this comment.
If you "stop" the mysql service and run
./manage.py monitors --service olympia_databaseThe command raises here as it is a check. we want to return the error object which means that check should not raise.
In general, this django check should also not raise but return the error object.
KevinMind
force-pushed
the
addons-15248-fix
branch
from
825fa3c to
48645c7
Compare
- Updated `docker-compose.yml` to improve health checks for worker and web services, ensuring they are monitored effectively. - Introduced a new management command `monitors.py` to check the health of various services, including database and web. - Updated `Makefile-docker` to replace healthcheck with monitors - Updated `initialize.py` to ensure the database is running before proceeding with initialization. - Removed the obsolete `healthcheck.py` script in favor of the new monitoring approach. - Added tests for the new monitoring functionality to ensure reliability. TMP: better tests
KevinMind
force-pushed
the
addons-15248-fix
branch
from
48645c7 to
ceb393d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: mozilla/addons#15249
Relates to: mozilla/addons#15248
Description
the custom healthcheck.py script was returning false positives. Instead of fixing this scripts, it makes more sense to use the existing infrastructure we have from monitors to ensure services are running correctly.
Context
We can use a management command running a set of monitors on a timeout to verify service dependencies are up. This is useful in the initializion command itself that depdends on the database being up, and after the initialization command where we want to ensure several critical dependencies are up before letting make up return successfully.
Testing
With a running environment you can run
Any combination of these should result in a failure. You can adjust the attempts parameter or the services parameter to check different services.
Checklist
#ISSUENUMat the top of your PR to an existing open issue in the mozilla/addons repository.