[spark] Use batch predict API

[xyang16]

Description

Brief description of what this PR is about

• If this change is a backward incompatible change, why must this change be made?
• Interesting edge cases to note here

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.42%. Comparing base (bd7f4ca) to head (b1f9f5b).
Report is 2 commits behind head on master.

❗ Current head b1f9f5b differs from pull request most recent head 8dd5433

Please upload reports for the commit 8dd5433 to get more accurate results.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3242      +/-   ##
============================================
+ Coverage     64.26%   72.42%   +8.15%     
- Complexity     6455     7304     +849     
============================================
  Files           676      676              
  Lines         32443    32443              
  Branches       3417     3417              
============================================
+ Hits          20849    23496    +2647     
+ Misses        10016     7335    -2681     
- Partials       1578     1612      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.