Support pip install directly from GitHub

[jamesbraza]

With Python 3.12.7 and pip==24.3.1, the below does not work:

> pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=tokenizers
Collecting git+https://github.com/huggingface/tokenizers.git#subdirectory=tokenizers
  Cloning https://github.com/huggingface/tokenizers.git to /private/var/folders/5c/20jqnfqx4sv1_6_bdkf765cr0000gn/T/pip-req-build-b1vkwjey
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/tokenizers.git /private/var/folders/5c/20jqnfqx4sv1_6_bdkf765cr0000gn/T/pip-req-build-b1vkwjey
  Resolved https://github.com/huggingface/tokenizers.git to commit 6c15458868bc79dce134dc821c1c0359f3180f2f
ERROR: git+https://github.com/huggingface/tokenizers.git#subdirectory=tokenizers does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

It would be cool if tokenizers supported pip installing from GitHub to get the latest source code, e.g. for Python 3.13 support.

[ArthurZucker]

Yeah sounds interesting, will look into this!
Will do the release today!

[gau-nernst]

@jamesbraza I think you point to the wrong subdirectory. This should work

pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python

[jamesbraza]

Oh wow you are right, I did. Closing this as resolved, thanks @gau-nernst