Add Dolma Counter script #85
Conversation
This PR adds a new dolma processor that can be used to count the number of (whitespace delimited) tokens in a data source. You can point it at a directory and it will find all the `.jsonl.gz` files in it or it's subdirectories. It can be used from the root dir via `python -m licensed_pile.count` or from anywhere with `count-tokens-dolma`.
Thats just what |
|
You're right, it not really configurable in dolma tho https://github.com/allenai/dolma/blob/64886d9db15bd99acea9e28740ae20a510875dfb/python/dolma/core/parallel.py#L268 I think it fine to leave it as is lol, plus the |
|
Very cool, thank you for doing this. Since "token" is sort of an overloaded term, and since I'm suggesting you also report the decompressed text-only (non-json-overhead) byte count, perhaps we can call this something about "size statistics" instead of tokens. |
The PR also allows one to point the script at a single dolma shard.
This PR adds a new dolma processor that can be used to count the number of (whitespace delimited) tokens in a data source.
You can point it at a directory and it will find all the
.jsonl.gzfiles in it or it's subdirectories.It can be used from the root dir via
python -m licensed_pile.countor from anywhere withcount-tokens-dolma.One weird aspect is that it uses prefixed like giga or tera "tokens" instead of a billion, a trillion, etc.