Download forge modules in parallel #284
Conversation
|
Looks like my editor changed a bunch of markdown stuff too with the vscode markdown plugin. |
|
Waiting for CLA signature by @logicminds @logicminds - We require a Contributor License Agreement (CLA) for people who contribute to Puppet, but we have an easy click-through license with instructions, which is available at https://cla.puppet.com/ Note: if your contribution is trivial and you think it may be exempt from the CLA, please post a short reply to this comment with details. http://docs.puppet.com/community/trivial_patch_exemption.html |
|
Not sure what is wrong the the CLA bot but I already signed so I can't sign twice. Also rubocop is failing without my changes. |
logicminds
force-pushed
the
parallel
branch
2 times, most recently
from
cbf42d7
to
374160f
Compare
Codecov Report
@@ Coverage Diff @@
## master #284 +/- ##
==========================================
+ Coverage 44.88% 45.62% +0.74%
==========================================
Files 11 11
Lines 782 789 +7
==========================================
+ Hits 351 360 +9
+ Misses 431 429 -2
Continue to review full report at Codecov.
|
logicminds
force-pushed
the
parallel
branch
2 times, most recently
from
95aad7d
to
bd7cb75
Compare
|
CLA signed by all contributors. |
|
This look really promising! :) What needs to be done to get this merged? It looks like the failing test is due to a Travis error and unrelated to the code changes. |
|
Probably just a rebase, and a slight code change to accommodate older puppet. I'll fix this and ping the maintainers. |
* previously only the git repositories were downloaded
in parallel. This saved a ton of time but forge modules
were still downloaded in serial.
* This adds the same parallel download feature of the repositories
to the forge modules.
* Refactors a few methods so they can be run in parallel.
* Extracts repository download method into more generic method
so we can reuse it with forge downloads.
* Typical speed improvments will be at least 400% with default of 10 threads.
The more modules defined the better it gets.
* Moves all methods to the fixtues helper module
* remove condition logic to or assignments
* move fixture references to cached value methods
* moves download logic to individual methods
* Forces rubocop to version 0.49
* previously anybody with rubocop gem younger than 0.49
would have issues with rubocop tests. This was due to needing
features in 0.49 and not enforcing that version.
This improves the forge module download speed by at least 400% when multiple modules are specified in fixtures.yml. Git repositories were previously already using parallel downloads so this only fixes modules.
This also refactors the fixtures code by adding almost everything into a method and placed inside the fixtures module spec module.