wheels for ARM64 Linux

[dalberto]

When attempting to install tiktoken in a the python:3.11-slim Docker container running in Docker for Mac, I get an error about a missing rust compiler( error: can't find Rust compiler...).

I'm not familiar with the Rust toolchain but I believe it should be possible to compile for aarch64 by add the following:

rustup target add aarch64-unknown-linux-gnu in pyproject.toml.

As a workaround, I'm just building from source, but it'd be super nice to have pre-built wheels.

My workaround:

export PATH="/root/.cargo/bin:$PATH"
apt-get update && apt-get install -y --no-install-recommends curl gcc &&  curl https://sh.rustup.rs -sSf | sh -s -- -y && apt-get install --reinstall libc6-dev -y

[hauntsaninja]

Thanks for the suggestion, but not planning on providing aarch64 wheels for now. In order to build and test aarch64 wheels in Github Actions, I'd need to use emulation, which is super slow and a pain to setup.

I'll leave this issue open for a bit, if this proves to be a popular request I might reconsider.

[sjrl]

We would definitely appreciate this as well for use in our docker images in the Haystack library

[aliencaocao]

+1, need the wheel

[frankdilo]

+1, without a wheel our team can't easily install this library in the local dev environment, making it a pain to use.

[kwood]

+1 on having the wheel! Developing on an M1 mac w/ Docker becomes difficult without it.
I didn't want to include the rust toolchain in my final docker image, so my (somewhat painful) workaround is to use a two-stage build like this:

FROM python:3.11-slim-bullseye as builder

# Tiktoken requires Rust toolchain, so build it in a separate stage
RUN apt-get update && apt-get install -y gcc curl
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && apt-get install --reinstall libc6-dev -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN pip install --upgrade pip && pip install tiktoken

FROM python:3.11-slim-bullseye
... 
# Copy pre-built packages from builder stage
COPY --from=builder /usr/local/lib/python3.11/site-packages/ /usr/local/lib/python3.11/site-packages/
# Now install packages as normal — tiktoken will already be installed and skipped if present in requirements.txt
RUN pip install --upgrade pip && pip install -r requirements.txt
...

[aiastia]

RUN pip install --upgrade pip && pip3 install tiktoken:
2318
#28 226.3 running build_rust
2319
#28 226.3 cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib
2320
#28 226.3 Updating crates.io index
2321
#28 226.3 error: cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib failed with code -9
2322
#28 226.3 [end of output]

[suanmiao]

+1 on needing the wheel, tiktoken is essential for many libraries and it avoids people from building docker image locally.

[twodotsjeremy]

+1 on the wheel as well! Appreciate any prioritization.

[hauntsaninja]

There's enough interest here that this is definitely something we should make happen. For various reasons, I've been short on time recently so can't commit to a timeline — but if someone opens a PR I will review and merge.

I copied over the QEMU setup I have from a different C project I maintain in #54 ; unfortunately it looks like it didn't just work. Maybe it's obvious to someone here what's wrong?

[jongwook]

@hauntsaninja It looks like a SIGKILL since it says exit code -9; is it possible that the runner's system memory was not enough? Could we try with a larger runner?

[aiastia]

Hello, I want to run tiktoken on arm64 server through docker, what should I do wrong now I tried #23 (comment)

But #23 (comment)

[messense]

I copied over the QEMU setup I have from a different C project I maintain in #54 ; unfortunately it looks like it didn't just work. Maybe it's obvious to someone here what's wrong?

I think it might just need to set a env var CARGO_NET_GIT_FETCH_WITH_CLI="true" because of rust-lang/cargo#10583

[Peins]

Me too.

[hauntsaninja]

I've released 0.3.1 along with aarch64 wheels. Thanks @messense for the cargo environment variable fix!