Resource health check polling (cloud traffic concerns)

[mitchdenny]

In #5601 we reduce the period for health check execution. We go from 30 seconds to 5 seconds. Some cloud hosted resources could be more expensive for health check intervals.

Do we need to introduce a Period property on the HealthCheckAnnotation and then improve the logic in ResourceHealthCheckScheduler to enable or disable a check based on whether the per resource internal has been reached. We might want to have two intervals, one for startup and one for normal running with the AddXYZResource method able to set the sensible default.

This would make the scheduler more complex.

[mitchdenny]

/cc @davidfowl

[mitchdenny]

Moving this to Backlog. PR #5870 should address most of the issues. We can look at this complete back-off if necessary down the track.

[Alirexaa]

@davidfowl, this issue in 9.0.0-rc.1 caused Meilisearch container logs to get noisy.

Can we solve this issue in 9.*?

We should reopen this issue.