add healthcheck support for http and smtp

[ap-wtioit]

this enables the user to configure a healthcheck for containers that can
be used to automatically restart proxies resolving to no longer active
ips due to PRE_RESOLVE

Issue description

on our longer running instances we discovered that after a while we couldn't reach proxied services anymore from odoo. we tracked it down to the docker-whitelist configs using PRE_RESOLVE: 1 for cloud services that are not always using the same ip for deployment (e.g. accounts.google.com) the ip changed after a few weeks while the running proxy still resolved to the old ip which was no longer serving the service.

New behaviour

with this change you can configure the required healthcheck for the proxied service type (http/smtp) that now enables docker to check the health of the proxy container. combined with https://github.com/willfarrell/docker-autoheal this enables us to automatically restart those containers no longer resolving to the correct ip.

We tested this for 4 months (and improving it in the meanwhile) on our prod system so it seems a worthy addition for us.

To me it looks like this also could be the foundation for fixing #5 (if the reporter confirms they are also using a target service with changing IP addresses) if we add an SQL healthcheck as well.

info @wt-io-it

[joao-p-marques]

This looks like a good addition! Thanks 👍

Can you please check the comments?

[yajo]

It looks much better! A couple of comments though.

[yajo]

I don't know if you knew this, but plumbum supports these kind of constructs out of the box (it would be equivalent (except the strip() part)):

Suggested change

	def _healthcheck(*args, **kwargs):
	args = ("-f", "tests/healthcheck.yaml") + args
	return docker_compose(*args, **kwargs)
	def _get_container_id(service_name):
	return _healthcheck("ps", "-q", service_name).strip()
	_healthcheck = docker_compose["-f", "tests/healthcheck.yaml"]
	_get_container_id = _healthcheck["ps", "-q", service_name]

[ap-wtioit]

_get_container_id = _healthcheck["ps", "-q", service_name]

does not work, because this syntax doesn't support args:

 NameError: name 'service_name' is not defined

[yajo]

Ah yes, sorry, my butter fingers. It should be _get_container_id = _healthcheck["ps", "-q"] instead, and then later called as _get_container_id(service_name).strip()

[ap-wtioit]

i changed it to

_healthcheck["ps", "-q"]

[joao-p-marques]

This looks great! Just (another) couple of comments 😄

[pedrobaeza]

I'm checking this now (after a while without seeing here). Is it everything up to date to check?

[ap-wtioit]

I rebased the previous 3 commits, and cherry-picked the improvements we made to our internal version in the last 3 years:

• adding timeouts for checking availability of services (in case the port becomes firewalled, e.g. when the service gets a new ip and the old ip is now unused and protected by a firewall)
• when using PRE_RESOLVE, check if the target still resolves to the same IP (so the container gets restarted when the service gets a new ip)
• detect load balancing ip addresses (e.g. when we get a new ip with every request) and skip the PRE_RESOLVE healthcheck from then on, preventing constantly restarting the container in this cases (the http / smtp healthchecks still work for those cases)

[ap-wtioit]

@pedrobaeza i think i finally made it work. Not at all sure about the poetry stuff (it was pretty broken on Ubuntu 23.10 and python 3.11) in the end i disabled typed-ast in poetry lock ran poetry add --dev pytest-timeout (and manually moved it to [tool.poetry.dev-dependencies]) ran poetry lock and checked with poetry install