Skip to content

Commit

Permalink
Clarify language models in README (#203)
Browse files Browse the repository at this point in the history
  • Loading branch information
logankilpatrick authored Jan 30, 2024
1 parent 3ee6c35 commit db5bda9
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ If you work at OpenAI, make sure to check the internal documentation or feel fre

## What is BPE anyway?

Models don't see text like you and I, instead they see a sequence of numbers (known as tokens).
Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens).
Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable
properties:
1) It's reversible and lossless, so you can convert tokens back into the original text
Expand Down Expand Up @@ -128,4 +128,3 @@ setup(

Then simply `pip install ./my_tiktoken_extension` and you should be able to use your
custom encodings! Make sure **not** to use an editable install.

0 comments on commit db5bda9

Please sign in to comment.