Add setup for multi node/multi-machine testing #2044

notbdu · 2019-11-15T03:27:20Z

What this PR does / why we need it:
Modifies scripts/vagrant to support both single and multi node testing. Multi node testing allows us to compare feature branches to release for both performance/stability testing.

Special notes for your reviewer:
This also adds additional environment variables to promremotebench's config.
PROMREMOTEBENCH_SECONDARY_TARGET
PROMREMOTEBENCH_SECONDARY_QUERY_TARGET

Does this PR introduce a user-facing and/or backwards incompatible change?:

NONE

Does this PR require updating code package or user-facing documentation?:

NONE

codecov · 2019-11-15T03:44:29Z

Codecov Report

Merging #2044 into master will increase coverage by 6.6%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #2044     +/-   ##
========================================
+ Coverage    65.8%   72.4%   +6.6%     
========================================
  Files        1001    1007      +6     
  Lines       85967   86460    +493     
========================================
+ Hits        56582   62614   +6032     
+ Misses      25300   19676   -5624     
- Partials     4085    4170     +85

Flag	Coverage Δ
#aggregator	`82% <0%> (+1.6%)`	⬆️
#cluster	`85.6% <0%> (+19%)`	⬆️
#collector	`64.8% <0%> (ø)`	⬆️
#dbnode	`79.8% <0%> (+3.9%)`	⬆️
#m3em	`73.2% <0%> (+0.9%)`	⬆️
#m3ninx	`73.9% <0%> (+2.9%)`	⬆️
#m3nsch	`51.1% <0%> (+22.7%)`	⬆️
#metrics	`17.6% <0%> (ø)`	⬆️
#msg	`74.9% <0%> (ø)`	⬆️
#query	`68.4% <0%> (+14.7%)`	⬆️
#x	`83.1% <0%> (+7.7%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfe741b...b6d0dfa. Read the comment docs.

robskillington · 2019-11-18T15:24:33Z

Great work, looking forward to testing our latest changes on HEAD vs last release (0.14.2).

arnikola

Bunch of nits, but looks good generally and happy to approve if you'd prefer to deal with them async

arnikola · 2019-11-15T18:47:17Z

scripts/vagrant/multi/README.md

+- kubernetes (using kind)
+- etcd (single node)
+- m3db operator
+- m3db node (single node)


nit: Should this be multiple nodes?

arnikola · 2019-11-15T22:14:53Z

scripts/vagrant/single/README.md

@@ -0,0 +1,71 @@
+# Vagrant
+
+This allows you to run a Kubernetes environment on a single box using vagrant to provision a VM either local or in a cloud environment.


nit: This runs a Kubernetes environment on a single box, using vagrant to provision a VM either locally, or in a cloud environment.

arnikola · 2019-11-15T22:16:11Z

scripts/vagrant/single/README.md

+
+# GCP setup
+
+If you authorized with `gcloud` you can use `~/.ssh/google_compute_engine` as your SSH key.


Nit: "After authorizing with gcloud, use ~/.ssh/google_compute_engine as the SSH key."

arnikola · 2019-11-15T22:18:55Z

scripts/vagrant/provision/rolling_restart_dbnodes.sh

+
+set -xe
+
+# Perform rollign restarts of db nodes forever.


nit: rolling

arnikola · 2019-12-05T18:48:33Z

scripts/vagrant/multi/README.md

+# Vagrant
+
+This allows you to run a Kubernetes environment on multiple boxes using vagrant to provision VMs either locally or in a cloud environment.
+Let's us compare a feature branch to release by specifying a `FEATURE_DOCKER_IMAGE` env variable.


nit:

This allows comparisons between a feature branch (specified by a `FEATURE_DOCKER_IMAGE` environment variable), and `latest`.

arnikola · 2019-12-05T19:17:40Z

scripts/vagrant/multi/README.md

+
+# Requirements
+Setup vagrant azure provider via [docs](https://github.com/Azure/vagrant-azure).
+Or altertnatively set up google provider via [docs](https://github.com/mitchellh/vagrant-google).


nit: alternatively

scripts/vagrant/multi/README.md

arnikola · 2019-12-05T19:19:48Z

scripts/vagrant/multi/Vagrantfile

+        # For heavy benchmarks, use pre-emptible n1-standard-64:
+        #google.machine_type = "n1-standard-64"
+        #google.preemptible = true
+        #google.auto_restart = false
+        #google.on_host_maintenance = "TERMINATE"


Can we key this on an env variable?

arnikola · 2019-12-05T19:23:07Z

scripts/vagrant/multi/start_vagrant.sh

+    fi
+
+    # Create ingress rules if not already exists.
+    MAYBE_M3COORDINATOR_INGRESS=$(gcloud --project=studious-saga-237001 compute firewall-rules list 2> /dev/null | tail -n +2 | awk '{ print $1 };' | grep default-allow-m3coordinator) || true


More of a question, what is studious-saga-237001?

It's the "Hosted Project" gcloud project, it seems to be the only one not attributed to a chronosphere environment.

Is it ok to expose this?

Yea, needed to expose this for traffic to be allowed from the node running the benchmarker and the coordinators. It's for testing anyways.

scripts/vagrant/multi/start_vagrant.sh

…kube.

…unt.

arnikola · 2020-01-07T17:11:37Z

src/dbnode/storage/bootstrap/bootstrapper/fs/source.go

@@ -112,6 +112,9 @@ func newFileSystemSource(opts Options) bootstrap.Source {
 	}
 	s.newReaderPoolOpts.alloc = s.newReader

+	s.log.Info("newFileSystemSource BoostrapIndexNumProcessors",


Think this may have snuck in from the other PR

arnikola · 2020-01-07T17:12:00Z

src/dbnode/runtime/runtime_options.go

@@ -46,7 +46,7 @@ const (
 	// attempt to divide a mutable index block into when flushing the index
 	// block. The fewer the blocks the better for searches, but the more memory
 	// intensive it is to build at runtime.
-	DefaultFlushIndexBlockNumSegments = 4
+	DefaultFlushIndexBlockNumSegments = 1


Think this may have snuck in from the other PR

arnikola

LGTM

notbdu requested review from robskillington and arnikola November 15, 2019 03:27

robskillington changed the title ~~Add setup for multi node/multi machine testing.~~ Add setup for multi node/multi machine testing Nov 18, 2019

robskillington changed the title ~~Add setup for multi node/multi machine testing~~ Add setup for multi node/multi-machine testing Nov 18, 2019

notbdu force-pushed the bdu/multi-machine-tests branch 2 times, most recently from 4dced99 to 89cf1b0 Compare November 19, 2019 08:04

arnikola approved these changes Dec 5, 2019

View reviewed changes

notbdu added 11 commits December 5, 2019 17:37

Add setup for multi node/multi machine testing.

d416e6b

Use env var for feature docker image

7608ed7

Use string delimited array of target write/query URLs.

e0258c8

Snaps docker installation is broken. Escape repo url.

db24361

Make tests work for azure.

7b3fa2d

Modify azure resource grouping. Use native port forwarding w/ docker/…

1f0fe08

…kube.

Add group nums to vm name.

9bcc9cb

Use storage optimized vms. Move docker data root path to temp data mo…

033eb94

…unt.

Fix docker mount setup.

06626c8

Change instance type and location.

7cae476

Wait until bootstrapped before killing next node and other minor fixes.

5ed21c5

notbdu force-pushed the bdu/multi-machine-tests branch from 358a450 to 5ed21c5 Compare December 5, 2019 22:38

save

957c262

Address PR comments.

f312e8a

arnikola reviewed Jan 7, 2020

View reviewed changes

notbdu added 3 commits January 7, 2020 12:16

Remove.

1d96f9e

Unremove.

609d418

Generate

0329975

arnikola approved these changes Jan 8, 2020

View reviewed changes

notbdu added 2 commits January 8, 2020 17:55

Merge branch 'master' into bdu/multi-machine-tests

4e01f63

Merge branch 'master' into bdu/multi-machine-tests

b6d0dfa

notbdu merged commit a55d39c into master Jan 9, 2020

notbdu deleted the bdu/multi-machine-tests branch January 9, 2020 22:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add setup for multi node/multi-machine testing #2044

Add setup for multi node/multi-machine testing #2044

notbdu commented Nov 15, 2019 •

edited

Loading

codecov bot commented Nov 15, 2019 •

edited

Loading

robskillington commented Nov 18, 2019

arnikola left a comment

arnikola Nov 15, 2019

arnikola Nov 15, 2019

arnikola Nov 15, 2019

arnikola Nov 15, 2019

arnikola Dec 5, 2019

arnikola Dec 5, 2019

arnikola Dec 5, 2019

arnikola Dec 5, 2019

notbdu Jan 7, 2020

arnikola Jan 8, 2020

notbdu Jan 9, 2020

arnikola Jan 7, 2020

arnikola Jan 7, 2020

arnikola left a comment

		@@ -0,0 +1,71 @@
		# Vagrant

		This allows you to run a Kubernetes environment on a single box using vagrant to provision a VM either local or in a cloud environment.


		# GCP setup

		If you authorized with `gcloud` you can use `~/.ssh/google_compute_engine` as your SSH key.

Add setup for multi node/multi-machine testing #2044

Add setup for multi node/multi-machine testing #2044

Conversation

notbdu commented Nov 15, 2019 • edited Loading

codecov bot commented Nov 15, 2019 • edited Loading

Codecov Report

robskillington commented Nov 18, 2019

arnikola left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnikola left a comment

Choose a reason for hiding this comment

notbdu commented Nov 15, 2019 •

edited

Loading

codecov bot commented Nov 15, 2019 •

edited

Loading