Process files in parallel to improve performance

[akoeplinger]

Not sure if you want to take it, but this improved performance for me as well (i.e. it now saturates all my four cores instead of only one).

Sample improvement for our website (both timings with PR #112 applied) on my four core machine:

before	after
real 1m26.940s	real 0m25.262s
user 1m24.883s	user 1m28.469s
sys 0m1.166s	sys 0m2.610s

[gjtorikian]

Yesssssssss I ❤️ parallelization.

Is this really all it takes? It seems so simple.

[akoeplinger]

Yep, at least the tests pass and it resulted in the same errors on my website. Props to the parallel gem for making this a breeze 🎉

[akoeplinger]

Only problem could be if there's some shared state that the threads try to access/modify concurrently, but from what I glanced this is not the case?

[gjtorikian]

One suggestion I would make: I don't like libraries I write (that depend on other libraries) to disregard their options.

Could you set up a way to pass options along to Parallel, similar to what's being done with Typhoeus? That way any consumer of this can pass along in_processes or in_threads to their heart's content. I don't want html-proofer blocking that.

If you get that working, docs + tests for it would be grand! Otherwise I can pick it up. Thanks!

[akoeplinger]

Hmm, I tried to add it, but I think something doesn't work, as I can't even pass a different Typhoeues option to the library. Output on master without any changes:

$ htmlproof _site --followlocation=false
/home/alexander/.rvm/gems/ruby-2.1.1/gems/mercenary-0.3.4/lib/mercenary/program.rb:30:in `go': invalid option: --followlocation=false (OptionParser::InvalidOption)
    from /home/alexander/.rvm/gems/ruby-2.1.1/gems/mercenary-0.3.4/lib/mercenary.rb:22:in `program'
    from /home/alexander/dev/html-proofer/bin/htmlproof:10:in `<top (required)>'
    from /home/alexander/.rvm/gems/ruby-2.1.1/bin/htmlproof:23:in `load'
    from /home/alexander/.rvm/gems/ruby-2.1.1/bin/htmlproof:23:in `<main>'
    from /home/alexander/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `eval'
    from /home/alexander/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `<main>'

[gjtorikian]

Yeah, I thought as much. Typhoeus is real picky about its options. Gonna merge this and work on the options in a separate PR.

[doktorbro]

I love it.

[kansaichris]

So by default this will use two processes?

[gjtorikian]

By default it should use whatever Parallel's defaults are.

[kansaichris]

Hm, okay.