Add a way to count tokens without encoding the whole text; improve performance; fix bugs#38

Open

niieani wants to merge 5 commits intolatitudegames:masterfrom niieani:bb/count-tokens

+5,474-3,783

Commits on Apr 15, 2023

Commits on Apr 16, 2023