Improve che.isCheServerReady() method (flapping route/ingress issue)

[benoitf]

From chectl created by rhopp: che-incubator/chectl#145

It turns out, that once k8s ingress is created it could be flapping between two states for some time -

• correctly redirecting to the service
• returning 503

This is causing some problems in our Happy Path test scenario:
We first start server by chectl server:start and then (pretty immediately) we are starting workspaces by chectl workspace:start --devfile=<devfile>.
What's happening?

• chectl server:start starts the server. In the step Che status check it calls <cheurl>/api/system/state to verify che server is up and running. This succeeds
• The same call (<cheurl>/api/system/state) is part of pre-flight checks for workspace:start command, but because the route/ingress flapping issue this fails.

We are implementing a workaround in our tests to overcome this issue (calling the ingress endpoint every second and waiting for multiple subsequent successful return codes), but we were thinking if it wouldn't be better to implement this logic into che.isCheServerReady() directly.

WDYT @l0rd @benoitf ?

[benoitf]

Nice description of the flapping issue is here: redhat-developer/rh-che#479 (comment)
At that time we were blaming openshift starter clusters, but it seems this problem is more general (appearing also in k8s - minikube)

[benoitf]

looks like there should be smth. similar to CHE_WORKSPACE_SERVER_PING__SUCCESS__THRESHOLD , but for chectl Che server ingress / route (configurable property, with number of sequential successful pings to server after which it is treated as available)

[benoitf]

@rhopp yes we could introduce a polling check parameter to increase the rate on calls on remote API.
Like wrapping axios calls to a polling-aware call.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.