Add Parallel spec rake tests, based on category with better output. #2671

shanear · 2017-07-20T18:29:54Z

The tests are parallelized by category, which makes for more sensible
output. The output is also all printed to the screen only when those
tests are finished. So there is no confusing overlapping output.

All unit tests are run as one category (a unit test is defined as
anything that isn't a feature test).

The feature tests are categorized by top level subfolders under
/spec/feature. All specs not in a subfolder will be part of the
other category.

The test categories can also be run seperately,
by seperate docker containers. They do not all have to be run at once.

Commands added

rake spec:parallel:setup - sets up the DBs needed for parallel testing
rake spec:parallel - runs all the different spec categories in parallel
rake spec:parallel:CATEGORY - runs specs from one category, but with parallel configurations set
rake spec:CATEGORY - runs specs from one category`

Test plan

Tested locally and verified Simplecov
10x successful Travis runs in a row
5x successful local runs in a row

Notes

The .travis.yml file could be cleaned up, but I don't want to invest to much into that because well probably be moving to jenkins soon.

connects #2458

The tests are parallelized by category, which makes for more sensible output. The output is also all printed to the screen only when those tests are finished. So there is no confusing overlapping output. All unit tests are run as one category (a unit test is defined as anything that isn't a feature test). The feature tests are categorized by top level subfolders under `/spec/feature. All specs not in a subfolder will be part of the `other` category. The test categories can also be run seperately, by seperate docker containers. They do not all have to be run at once. `rake spec:parallel:setup` - sets up the DBs needed for parallel testing `rake spec:parallel` - runs all the different spec categories in parallel `rake spec:parallel:CATEGORY` - runs specs from one category, but with parallel configurations set `rake spec:CATEGORY` - runs specs from one category` - Tested locally - Remove parallel-tests gem - add documentation - figure out why one reader test is consistently failing when parallelized - integrate into the Jenkins job

joofsh · 2017-07-20T18:35:05Z

config/database.yml

@@ -78,7 +78,7 @@ test:
  <% if ENV['POSTGRES_PASSWORD'] %>
  password: <%= ENV['POSTGRES_PASSWORD'] %>
  <% end %>
-  database: caseflow_certification_test<%= ENV['TEST_ENV_NUMBER'] %>
+  database: caseflow_certification_test<%= ENV['TEST_SUBCATEGORY'] %>


should we update the Redis config too? It looks like they'd all be hitting the same redis namespace which could cause clashes (an issue with the existing parallel tests too)

I am updating the cache config here: https://github.com/department-of-veterans-affairs/caseflow/pull/2671/files#diff-639e7393aa097999e816b53faa94bab4L11

our tests don't actually run against redis. We are using Rails file cache. Which is sort of a stubbed cache.

Interesting, I just noticed that for the first time. I wonder why we use file_store for tests? Couldn't we use memstore to both speed up the tests & more closely mirror production? of course, this would be outside the scope of this PR but just wondering your thoughts

Yeah i agree with you on both points. It's worth considering upgrading our test caching mechanism as well as it being outside the scope of this PR.

Also I found this fun little guy:
https://github.com/department-of-veterans-affairs/caseflow-commons/pull/108/files

joofsh · 2017-07-20T18:35:50Z

lib/tasks/spec.rake

+
+namespace :spec do
+  feature_spec_folders = Dir.entries("./spec/feature").select do 
+    |file| !['.', '..'].include?(file) && File.directory?(File.join("./spec/feature", file))


can we move the |file| to the line above to be more idiomatic

not sure how this happened. I'm very embarrassed.

joofsh · 2017-07-20T18:38:38Z

lib/tasks/spec.rake

+      task spec_name do
+        envs = "export TEST_SUBCATEGORY=#{spec_name};"
+
+        run_command_and_print_output("#{envs} rake spec:#{spec_name}")


shouldn't the rake spec:reader itself set it's own TEST_SUBCATEGORY?

I originally did that but figured it was more stable to set the env variable out of process. I could move it back in.

joofsh · 2017-07-20T18:43:12Z

lib/tasks/spec.rake

+require 'rspec/core/rake_task'
+
+def run_command_and_print_output(command)
+  output_stream = open("|#{command}", "r")


You previously created a ShellCommand class for this sort of thing. Do we want to use that here to standardize? It looks like its run method should be usable here, but if not, can we move this logic there so it can be reused?

gooooood catch. I will move this there.

joofsh · 2017-07-20T18:43:20Z

lib/tasks/spec.rake

+  output = ""
+  output_stream.each { |line| output << line; print "."; $stdout.flush; }
+  puts output
+end


can you add comments explaining how this method works? I don't quite grok is. Does the output from the open command immediately return to output_stream as an array?

joofsh · 2017-07-20T18:44:34Z

lib/tasks/spec.rake

+    |file| !['.', '..'].include?(file) && File.directory?(File.join("./spec/feature", file))
+  end
+
+  spec_names = feature_spec_folders + ["other", "unit"]


should this and feature_spec_folders be constants?

joofsh · 2017-07-20T18:46:23Z

.simplecov

@@ -9,6 +9,6 @@ if ENV["RAILS_ENV"] == "test"
    add_filter "lib/tasks"
  end

-  SimpleCov.command_name "Test Run" + (ENV["TEST_ENV_NUMBER"] || "")
+  SimpleCov.command_name ENV["TEST_SUBCATEGORY"] || "all"


can you extend your test plan section. Did you run the new test setup and verify simplecov output? Did you verify the output shows up in travis? etc

Im not intending to set this up in travis. Just jenkins.

I did veryify simplecov is working, but I should write that down too.

joofsh · 2017-07-20T18:48:09Z

config/environments/test.rb

@@ -8,7 +8,7 @@
  # and recreated between test runs. Don't rely on the data there!
  config.cache_classes = true

-  cache_dir = Rails.root.join("tmp", "cache", "paralleltests#{ENV['TEST_ENV_NUMBER']}")
+  cache_dir = Rails.root.join("tmp", "cache", "test_#{ENV['TEST_SUBCATEGORY']}")


At a high level can you describe how this setup works? Are these all running against the same instance of the application? It looks like all the parallel tests would hit on the same port. Is this problematic? Does that code need to be updated too possibly?

They are different files where the Rails file_store cache is stored. I'm just configuring the tests to use different files for each test category here.

joofsh

Looks great in general, thanks for doing this! +1 to cleaning up & removing the parallel tests gem 👍

joofsh

A couple things worth fixing, but I don't want to block this PR since you're going to run into merge conflict hell the longer this stays open 👍 🙏

…ns-affairs/caseflow into parallel_ci

Merge branch 'parallel_ci' of github.com:department-of-veterans-affairs/caseflow into parallel_ci

shanear · 2017-07-28T23:25:30Z

fire in the hole 🔥

shanear assigned joofsh Jul 20, 2017

shanear added the In Progress label Jul 20, 2017

joofsh reviewed Jul 20, 2017

View reviewed changes

joofsh suggested changes Jul 20, 2017

View reviewed changes

joofsh approved these changes Jul 21, 2017

View reviewed changes

Alan Ning and others added 24 commits July 22, 2017 17:42

Merge branch 'master' into parallel_ci

13b864d

test

73895bd

removing lint

03a2781

changed export command

0b76a0f

Merge branch 'master' into parallel_ci

e92ee78

Test travis parallel

f89e8eb

Take 2

7353417

Another fix

9a52744

tweaks

d1f5d94

More tweaks

4348e60

Tweak

0891dec

Another tweak

4422738

fix

36e9c91

more tweaks

324c16f

Merge branch 'parallel_ci' of https://github.com/department-of-vetera…

ebb365f

…ns-affairs/caseflow into parallel_ci

Merge branch 'parallel_ci' into feature/parallel_ci_alan

c33039c

test

860199b

test

1f67bf4

test

e47dd3f

test

a497ae5

test

61a2b8c

test

89c66f2

test

66a825b

test

0bcd304

Alan Ning and others added 26 commits July 28, 2017 09:16

saving work

b440bed

saving work

06c935a

saving work

aca6f43

saving work

4f1a25f

saving work

50ca831

saving work

56e7518

saving work

75c4dfb

saving work

ec62cbb

saving work

126edd2

saving work

66951db

fix

d21908e

Merge branch 'master' into parallel_ci

f7acff6

test coverage failure

4cb8f49

Merge branch 'feature/parallel_ci_alan' into parallel_ci

fad5fa7

remove parallel tests

018f7ed

Fix code coverage

3a5db90

Merge branch 'master' into parallel_ci

a16d330

More tweaking

f1ad55e

t push

fdbb57d

Merge branch 'parallel_ci' of github.com:department-of-veterans-affairs/caseflow into parallel_ci

fix

41e2852

more fixes

c3b0525

Fixes, build should pass

fc03eb4

linty

c9f5727

fix

50d1853

tweak

1803c55

Merge branch 'master' into parallel_ci

60ee52a

shanear merged commit 342ef00 into master Jul 28, 2017

shanear removed the In Progress label Jul 28, 2017

ghost deleted the parallel_ci branch August 5, 2017 14:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Parallel spec rake tests, based on category with better output. #2671

Add Parallel spec rake tests, based on category with better output. #2671

shanear commented Jul 20, 2017 •

edited

Loading

joofsh Jul 20, 2017

shanear Jul 20, 2017 •

edited

Loading

joofsh Jul 21, 2017

shanear Jul 27, 2017

joofsh Jul 20, 2017

shanear Jul 20, 2017

joofsh Jul 20, 2017

shanear Jul 20, 2017

joofsh Jul 20, 2017 •

edited

Loading

shanear Jul 20, 2017

joofsh Jul 20, 2017

joofsh Jul 20, 2017

joofsh Jul 20, 2017

shanear Jul 20, 2017

joofsh Jul 20, 2017

shanear Jul 20, 2017

joofsh left a comment

joofsh left a comment

shanear commented Jul 28, 2017

Add Parallel spec rake tests, based on category with better output. #2671

Add Parallel spec rake tests, based on category with better output. #2671

Conversation

shanear commented Jul 20, 2017 • edited Loading

Commands added

Test plan

Notes

Choose a reason for hiding this comment

shanear Jul 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joofsh Jul 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joofsh left a comment

Choose a reason for hiding this comment

joofsh left a comment

Choose a reason for hiding this comment

shanear commented Jul 28, 2017

shanear commented Jul 20, 2017 •

edited

Loading

shanear Jul 20, 2017 •

edited

Loading

joofsh Jul 20, 2017 •

edited

Loading