Remove sharding on functional tests #2104
Conversation
ssbarnea
added
the
test
* Remove sharding on functional tests Sharding tests was hacky approach for dealing with Travis build time limitations, one that caused lots of issues. In order to simplify the testing and ease the migration toward other CI system, we are better-off dropping the sharding feature (which also seems to prevent bumping pytest). * Remove most YAML anchors which prevented use of official syntax for travis file for "env" blocks (lists not dictionaries). * Consolidated newlines between jobs (readability) Note: This is more of a POC change as we will need other optimizations in order to assure that functional always runs under 50min time limit. Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
|
Great, but we really can't do that before the need to have builds running on Travis has disappeared, because all builds on Travis will fail by design. |
|
That is true and this is why it was marked as a draft. Still, there is something that could potentially make this real: if we manage to optimize the execution time. Currently functional takes around 60min and Travis limit is 50mins. If we cut it to ~40mins we can make it real (we need to have ~10min safety net). |
This comment has been minimized.
This comment has been minimized.
|
Ok, new idea. Since #2139 and #2137 show that we can run a whole bunch of Molecule in parallel I propose that we actually move forward with this attempt. We remove sharding altogether. For the functional test suite, I propose a marker ( EDIT: Further clarification: instead of running each functional job with shard 1, 2, 3 we would instead run each functional job against This might work! Then I can finally get on with #1811 and ride into the sunset ... |
Showing a smaller diff than ansible#2104
|
Hey, take a look at decentral1se@cdf902c. That's a much smaller diff with less unrelated changes for removing sharding. If we merged #2139 and #2137 then I could work from this sharding diff to refactor the Travis configuration to use |
|
FYI, plan has changed now since further experimentation. There are 200+ functional tests which must be run in serial execution mode. So, we can't afford to remove sharding even if we're running some tests in parallel mode (~25 tests only!). So, let's close this off? |
|
Closing because is not doable yet. |
Sharding tests was hacky approach for dealing with Travis build time
limitations, one that caused lots of issues.
In order to simplify the testing and ease the migration toward other
CI system, we are better-off dropping the sharding feature (which also
seems to prevent bumping pytest).