Ensure tokenizer sizers with truncation parameters count their overflow encodings

[Jeadie]

Bug fix

When determining the size of a chunk that would exceed a tokenizer's TruncationParams (when set, generally via a tokenizer.json's truncation key), it would ignore the Encoding::get_overflowing.

This PR fixes this issue by recursively computing the size of these overflows too.

---

Note: Not all Tokenizers have truncation, hence passing tests for Tokenizer::from_pretrained("bert-base-cased", None) (link).

• thenlper/gte-small has truncation parameters, but they are too large to be causing issues in unit testing.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.39%. Comparing base (f1f81e7) to head (424f395).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #433   +/-   ##
=======================================
  Coverage   99.38%   99.39%           
=======================================
  Files          11       11           
  Lines        1953     1974   +21     
=======================================
+ Hits         1941     1962   +21     
  Misses         12       12           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[benbrandt]

Thanks @Jeadie ! There is always more to learn about the tokenizers crate 😅

I'll work on getting this merged and released since I assume this bug is affecting you.