Improved e2e testing: implemented getDocumentTokens in IndexReaderUtils

[lintool]

Implemented getDocumentTokens, which returns a list of tokens - this makes testing easier than getDocumentVector.

[codecov]

Codecov Report

Merging #1521 (f8d1932) into master (6b24bbd) will increase coverage by 0.04%.
The diff coverage is 72.41%.

@@             Coverage Diff              @@
##             master    #1521      +/-   ##
============================================
+ Coverage     57.38%   57.43%   +0.04%     
- Complexity      982      988       +6     
============================================
  Files           166      166              
  Lines          9107     9136      +29     
  Branches       1231     1240       +9     
============================================
+ Hits           5226     5247      +21     
- Misses         3494     3498       +4     
- Partials        387      391       +4     

Impacted Files	Coverage Δ	Complexity Δ
.../main/java/io/anserini/index/IndexReaderUtils.java	76.79% <72.41%> (-0.62%)	44.00 <6.00> (+6.00)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b24bbd...f8d1932. Read the comment docs.

[stephaniewhoo]

lgtm, this makes doc tokens clearer.