mgpu predictor using explicit offsets

[rongou]

Alternative approach to #4437. Uses explicit offsets of slide over the out prediction vector. I feel this is slightly cleaner, but I'm ok with either approach.

@canonizer @RAMitchell @sriramch

[RAMitchell]

I like this PR based on its simplicity. @rongou @sriramch can one of you please do an experiment comparing the performance of both of your PRs checking for large differences in performance. It sometimes hard to see exactly how many memory copies are occurring when using HostDeviceVector.

[sriramch]

@RAMitchell thanks for your review. this 'pr' can optimize away those copies for the batch prediction which the other 'pr' does. but, i think this 'pr' still has to work for smaller batch sizes. i have provided some comments to this effect earlier.

[rongou]

@RAMitchell @sriramch I ran some experiments on these two PRs using 1 billion rows, written to a tmpfs directory on a GCP VM with 4x T4 GPUs.

Timing of the PredictBatch() method:

This PR:
18.972 s
17.855 s
18.054 s

#4437:
20.842 s
20.123 s
19.931 s

Looks like this PR is slightly faster.

[rongou]

@canonizer @RAMitchell @sriramch @trivialfis all the comments have been addressed. Please take another look.

[sriramch]

rest lgtm...

[RAMitchell]

LGTM aside from one comment

[rongou]

@hcho3 looks like the build machine ran out of disk space?

[2019-05-09T02:17:46.935Z] tar: .m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.pom: Cannot write: No space left on device

[hcho3]

@rongou I will go ahead and expand disk space for the slave workers.

[hcho3]

@rongou @RAMitchell I think this PR should be part of 0.90, since it's a follow up to #4284. Can we merge this now?