[coordinator] [query] Add readiness probe for probing current consistency level achievability

[robskillington]

What this PR does / why we need it:

This adds a readiness probe that returns whether the cluster is available for reads and/or writes at the configured consistency levels. Useful if in an environment it takes some time for the process to achieve connectivity before being added behind a load balancer.

Special notes for your reviewer:

Does this PR introduce a user-facing and/or backwards incompatible change?:

NONE

Does this PR require updating code package or user-facing documentation?:

NONE

[ryanhall07]

it would be good to document how this readiness check works (ie. returns a 5xx if writes are not ready and you requested writes)

[robskillington]

Good point.

[ryanhall07]

wdyt about skipping the respective availability check if not requested? e.g skip checking write available if req.writes is false. my thinking is it would be unfortunate to block reads if only reads were requested and the write availability ran into an error.

[robskillington]

So these don't actually wait on anything it's just a check of current state (checking open connection count), not actually trying to do reachability so won't be any faster/slower or blocking if one is unhealthy vs another.

[linasm]

In general LGTM with a few readability nits.
But I think unit tests for session.WriteConsistencyLevel / session.ReadConsistencyLevel would be handy as the logics there is not that trivial.

[linasm]

Nit: I would extract this switch out of the loop and compute minReplicasRequired value in it (which would be topoMap.Replicas(), topoMap.MajorityReplicas(), or 1, respectively).
(not for the sake of performance, but for brevity and reduced nesting)

[linasm]

... and then, this would simply be:

Suggested change

	if !clusterAvailableForShard {
	return false, nil
	}
	if shardReplicasAvailable < minReplicasRequired {
	return false, nil
	}

[linasm]

Micro nit: invert then/else to avoid negation.

[linasm]

Same micro nit as above wrt !ready.

[robskillington]

I refactored the code so that the existing connection code (which uses same logic) is reused, and that's already covered by extensive code coverage thankfully.

[codecov]

Codecov Report

Merging #2976 (269215f) into master (421a988) will increase coverage by 0.5%.
The diff coverage is 51.4%.

@@           Coverage Diff            @@
##           master   #2976     +/-   ##
========================================
+ Coverage    71.8%   72.3%   +0.5%     
========================================
  Files        1078    1078             
  Lines      100253   99330    -923     
========================================
- Hits        72039   71881    -158     
+ Misses      23238   22470    -768     
- Partials     4976    4979      +3     

Flag	Coverage Δ
aggregator	75.8% <ø> (-0.1%)	⬇️
cluster	85.3% <ø> (ø)
collector	84.3% <ø> (ø)
dbnode	78.8% <46.1%> (+1.0%)	⬆️
m3em	74.4% <ø> (ø)
m3ninx	73.1% <ø> (-0.1%)	⬇️
metrics	19.9% <ø> (ø)
msg	74.0% <ø> (-0.1%)	⬇️
query	67.3% <55.5%> (+0.4%)	⬆️
x	80.2% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 421a988...269215f. Read the comment docs.