Fails with Parallel::DeadWorker: Parallel::DeadWorker on Travis

[fniephaus]

We use Travis to build, test and deploy our website and everything has been working fine until today.
Whenever a new Travis build runs, the html-proofer fails with a Parallel-related error:
https://travis-ci.org/squeak-smalltalk/squeak.org/builds/98519640#L237

How can we fix this?

[gjtorikian]

I suggest opening this up with Travis. A new release of HTML-Proofer hasn't gone out in a while so it's more likely that they changed something on their end. See if they made a change in the last few days, I guess. Sorry I cannot be more helpful!

As an aside, I love Smalltalk and I'm thrilled Squeak is using this project! 😁

[gjtorikian]

Wow. I'm working on a branch locally and just hit the same error:

6) Command test works with disable-external
     Failure/Error: expect(output).to match('successfully')
       expected "Running [\"LinkCheck\", \"ImageCheck\", \"ScriptCheck\"] on spec/html/proofer/fixtures/links/brokenLinkExternal.html on *.html... \n\n\nhtmlproof 2.6.1 | Error:  Parallel::DeadWorker\n" to match "successfully"
       Diff:
       @@ -1,2 +1,5 @@
       -successfully
       +Running ["LinkCheck", "ImageCheck", "ScriptCheck"] on spec/html/proofer/fixtures/links/brokenLinkExternal.html on *.html... 
       +
       +
       +htmlproof 2.6.1 | Error:  Parallel::DeadWorker

Huh.

[gjtorikian]

Got it. I think the in_processes method is creating a conflict somehow. It seems that restarting the Travis build / rerunning the test locally clears this up. I'm now not certain whether it's a random error or something to be fixed by Parallel, but nothing around this code has changed in Proofer recently, so I'm inclined to believe the issue is somewhere else.

[fniephaus]

Thanks for looking into this.
I figured the same. However, it didn't go away after restarting the build multiple times.
Any idea how to proceed with this?

BTW: this build had the same issue 3 months ago:
https://travis-ci.org/Katello/katello.org/builds/79476253#L154

[fniephaus]

I still haven't been able to resolve the problem.
And I'm still not sure if this related to parallel or html-proofer.
Check this out:
https://travis-ci.org/squeak-smalltalk/squeak.org/builds/101306515#L276

[fniephaus]

Hi @gjtorikian,

I'd like to reopen this issues, because I do believe that this is an html-proofer-related issue.

I tried to debug the problem, but didn't really have success.
But as far as I can tell, the problem must have been introduced with v2.6.0, see here:
https://travis-ci.org/squeak-smalltalk/squeak.org/builds/101312462
https://travis-ci.org/squeak-smalltalk/squeak.org/builds/101312937

We will use v2.5.2 for now and please feel free to add Squeak to your real life examples 😃

[fniephaus]

This reveals a problem with clean_content(string):
https://travis-ci.org/squeak-smalltalk/squeak.org/builds/101306515#L276

So maybe this commit introduced the problem: 6ab98af

[gjtorikian]

I just released 2.6.2 which I believe should fix this. Please let me know if the issue continues!

[gjtorikian]

Scratch that. 2.6.4 turns of Parallel processing by default. You can pass in your own options via the :parallel option.

My best guess is that after a certain number of files, the parallel processing stops. I can't reproduce it locally, but I do see it on our internal CI too. Turning off parallel processing completely works; setting in_processes to a lower number also works.

[fniephaus]

Works like a charm. Thanks!

[wjdp]

Getting the same as @fniephaus, see https://travis-ci.org/newtheatre/history-project/builds/157627379

Turning of parallelism, as expected task takes much longer (~35 mins). But then fails with the following, seems to be running out of resources.

/home/travis/build.sh: line 45: 3569 Killed bundle exec rake test

https://travis-ci.org/newtheatre/history-project/builds/157636834

Seems this project doesn't scale well with large sites, are there plans to address this?