feat: Implement o200k_base encoding and support gpt-4o

[chatanywhere]

Implement o200k_base encoding and support gpt-4o

[fernando-arellano]

Hello @tox-p thanks for all the support on maintaining this repo. Is there anything we can do to help you with this PR? we're relying on this library in our project and righ now we're looking to start using gpt4o but it seems the library doesn't have support yet for gpt4 o. please let us know if there is something we can do to help with this.

[jcrupi]

Checking in on this approval

[mkemmerz]

@tox-p Sorry to annoy you directly but any plan to review this PR?

[keyurgolani7878]

Hey, @tox-p Can you provide update on when are you planning to review the PR? I need gpt-4o encoding ASAP. Thanks!

[Grogdunn]

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

[keyurgolani7878]

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

I can't find this on maven central. Can you help me?

[Grogdunn]

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

I can't find this on maven central. Can you help me?

Strange, I've released this morning and I see in central portal: https://central.sonatype.com/artifact/net.optionfactory/jtokkit

[mkemmerz]

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

I can't find this on maven central. Can you help me?

Strange, I've released this morning and I see in central portal: https://central.sonatype.com/artifact/net.optionfactory/jtokkit

Not sure what dependency that is but not the one from this Github repo, this is the correct maven repo: https://mvnrepository.com/artifact/com.knuddels/jtokkit

[Grogdunn]

@mkemmerz of course that dependency has different groupId. It came from a fork with merged support for GPT-4o and GPT-4-turbo models. Nothing "strange", I've need of this functionality for some customers, so I've published myself. Here the fork https://github.com/optionfactory/jtokkit and thanks for the peole who made the PR.

[joesh36]

Any progress made to merge it to the main repo?

[joesh36]

Hey, @tox-p there is a plan to review this PR soon? we really need this encoding.
Thanks

[mkemmerz]

Thanks 👍