Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow Browser Usage, countTokens convenience method #33

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Allow Browser Usage, countTokens convenience method #33

wants to merge 1 commit into from

Conversation

wesmcouch
Copy link

Overview
  • Remove fs and path usage to allow usage of the encoder and decoder both inside a browser and in nodejs
  • Add a convenience method named countTokens that counts the number of tokens in a given string
  • Update the example and tests to include the new method countTokens

@wesmcouch
Remove fs and path usage to allow gpt-3-encoder to work in the browse…
98a8be5 
…r and in nodejs. Add countTokens function and tests.
@Clueed Clueed mentioned this pull request Feb 12, 2023
Open
@nem035
Copy link

nem035 commented Feb 28, 2023

FYI - I published a version to use (works in browser/node but no countTokens) until the maintainers merged: https://www.npmjs.com/package/@nem035/gpt-3-encoder

@arcticfly
Copy link

@nem035 looks like you can merge?

@astoilkov
Copy link

This PR also makes it work in Electron (not just the browser). Vite also has a problem with the code in master because it uses __dirname.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CheeseDurger CheeseDurger CheeseDurger approved these changes

@nem035 nem035 nem035 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wesmcouch @nem035 @arcticfly @astoilkov @CheeseDurger