Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelized Travis runs split test, coverage reports #5931

Closed
jywarren opened this issue Jun 21, 2019 · 10 comments · Fixed by #5954
Closed

Parallelized Travis runs split test, coverage reports #5931

jywarren opened this issue Jun 21, 2019 · 10 comments · Fixed by #5954
Labels
bug the issue is regarding one of our programs which faces problems when a certain task is executed discussion testing issues are usually for adding unit tests, integration tests or any other tests for a feature

Comments

@jywarren
Copy link
Member

so we get partial reports of everything, like our coverage being reported at 53%, which is actually only the coverage calculated from the system tests task: https://travis-ci.org/publiclab/plots2/jobs/548708932#L4077

In a similar vein, our PR Dangerbot posts can no longer access aggregate junit reports in Writing XML reports to /app/test/reports -- since we're running different tests in parallel in each of 5 tasks (and 5 different VMs).

I think we have to brainstorm on this -- mostly for our coverage reporting but also for the Dangerbot failure reporting, which is now erroring and not linking into any tests.

plots2/Dangerfile

Lines 32 to 44 in 21261a5

junit.parse "output.xml"
junit.failures.collect(&:nodes).flatten.each do |failure|
failure.nodes.each do |f|
f.match(/(test[a-z_\/]+.rb):([0-9]+)/) do |match|
source_path = match[1]
line = match[2]
if !source_path.nil? && !line.nil?
f = f.gsub(source_path + ':' + line, "<a href='https://github.com/#{github.pr_author}/plots2/tree/#{github.branch_for_head}/#{source_path}#L#{line}'>#{source_path}:#{line}</a>")
end
fail("There was a test failure at: #{f}")
end
end
end

@jywarren jywarren added bug the issue is regarding one of our programs which faces problems when a certain task is executed testing issues are usually for adding unit tests, integration tests or any other tests for a feature discussion labels Jun 21, 2019
@kaustubh-nair
Copy link
Member

We're facing a similar issue in mapknitter as well. Codecov reports a coverage of 40% which is only due to unit tests. I believe we need to find a way to accumulate all the reports and send them together to Codecov. Instead of sending 5 different reports
CC @alaxalves

@jywarren
Copy link
Member Author

jywarren commented Jun 25, 2019 via email

@alaxalves
Copy link
Member

Is there any shared storage of any kind between tasks?

On Tue, Jun 25, 2019 at 3:21 AM Kaustubh Nair @.***> wrote: We're facing a similar issue in mapknitter as well. Codecov reports a coverage of 40% which is only due to unit tests. I believe we need to find a way to accumulate all the reports and send them together to Codecov. Instead of sending 5 different reports CC @alaxalves https://github.com/alaxalves — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5931?email_source=notifications&email_token=AAAF6J27Q3QCPOXZOWNIHZTP4HBO7A5CNFSM4H2SKZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPJIMQ#issuecomment-505320498>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6JZAIPKOBSBPYI5VTMLP4HBO7ANCNFSM4H2SKZPQ .

I'll into it. I'm already digging this to Mapknitter, when I get it figured out there, I'll do it here as well.

@jywarren
Copy link
Member Author

Hm. Thank you both! Note in relation to this one, I'm using an env var in #5929 to make only one of the 5 tasks actually generate a Danger report!

@kaustubh-nair
Copy link
Member

@jywarren are we using coveralls for plots2?
If so, we can simply set COVERALLS_PARALLEL=true for merging the reports.
http://docs.coveralls.io/parallel-build-webhook

@kaustubh-nair
Copy link
Member

And set up a webhook

@kaustubh-nair
Copy link
Member

kaustubh-nair commented Jun 26, 2019

It would be easier to switch to Codecov. They auto merge test reports if we use the gem. I've set it up here publiclab/mapknitter#749
Thanks!

@jywarren
Copy link
Member Author

jywarren commented Jun 26, 2019 via email

@kaustubh-nair
Copy link
Member

Okay! I'll submit a PR if you'd like

@jywarren
Copy link
Member Author

jywarren commented Jun 26, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug the issue is regarding one of our programs which faces problems when a certain task is executed discussion testing issues are usually for adding unit tests, integration tests or any other tests for a feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants