[coordinator] Docker integration tests for prom-remote storage

[Antanukas]

What this PR does / why we need it:

In this PR docker integration test suite is implemented to cover Prom Remote Write backend use case from #3742.

Special notes for your reviewer:

For tests we setup:

1. etcd instance
2. coordinator admin
3. coordinator with Prom Remote Write backend
4. 2 aggregators
5. 2 prometheus instances that accept remote writes --enable-feature=remote-write-receiver

Coordinator has 2 endpoints setup prometheusraw and prometheusagg. Metrics are written to coordinator using remote write API http://m3coordinator01:7201/api/v1/prom/remote/write. There is a rollup rule that gets applied and metric gets renamed before sending to aggregated prometheus.

Both prometheus are queried and are expected to return a metric.

Does this PR introduce a user-facing and/or backwards incompatible change?:

None

Does this PR require updating code package or user-facing documentation?:

None

[codecov]

Codecov Report

Merging #3768 (7b4e367) into master (7b4e367) will not change coverage.
The diff coverage is n/a.

❗ Current head 7b4e367 differs from pull request most recent head 862d4c8. Consider uploading reports for the commit 862d4c8 to get more accurate results

@@          Coverage Diff           @@
##           master   #3768   +/-   ##
======================================
  Coverage    57.1%   57.1%           
======================================
  Files         552     552           
  Lines       62984   62984           
======================================
  Hits        35974   35974           
  Misses      23819   23819           
  Partials     3191    3191           

Flag	Coverage Δ
aggregator	63.2% <0.0%> (ø)
cluster	∅ <0.0%> (∅)
collector	58.4% <0.0%> (ø)
dbnode	60.8% <0.0%> (ø)
m3em	46.4% <0.0%> (ø)
metrics	19.7% <0.0%> (ø)
msg	74.3% <0.0%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b4e367...862d4c8. Read the comment docs.

[vpranckaitis]

Why are these being piped though jq docker container? Could it pipe through local jq instead (i.e.... | jq ...) like in other functions in this file?

[Antanukas]

I am not sure. My guess - to not force installation on JQ on a running machine. I copied (and removed some uneeded stuff) from other tests.

[vpranckaitis]

That seems reasonable. However, IIUC, other code in the same file uses local jq, e.g.

m3/scripts/docker-integration-tests/prom_remote_write_backend/utils.sh

Lines 50 to 51 in 36e3845

	ATTEMPTS=50 TIMEOUT=2 MAX_TIMEOUT=4 retry_with_backoff \
	"[[ \$(curl -sSf $host/api/v1/query?query=$metric_name | jq -r .data.result[0].value[1]) -gt 0 ]]"

So we still have to have jq on machine, with the additional complexity of running some of the jq commands through docker container.

[Antanukas]

ah yeah, actually I used local myself in couple of commands without thinking. Will use local here as well.