Running minilm model local only

[gerritvd]

I have model data for sentence-transformers/all-MiniLM-L6-v2 located in /mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125

I noticed that infinity seems to load hub data multiple times

• config.json
  • this seems to come from model_name_or_path
• the model data that includes config.json
  • this seems to come from SENTENCE_TRANSFORMERS_HOME env var

So my thinking was, I need to set:

SENTENCE_TRANSFORMERS_HOME=/mnt/models/hub

and

model_name_or_path=/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125

Final docker command:

docker run -e HF_HUB_OFFLINE=1 -e SENTENCE_TRANSFORMERS_HOME=/mnt/models/hub -e HF_MODEL=sentence-transformers/all-MiniLM-L6-v2 -v/home/$USER/.cache/huggingface/hub:/mnt/models/hub -p7997:7997 -it --rm --entrypoint=infinity_emb infinity:v0.0.23 --model-name-or-path=/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125

However, I'm presented with the following error:

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO     2024-02-06 01:06:17,642 infinity_emb INFO: model=`/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125` selected, using engine=`torch` and device=`None`                                                                                                                select_model.py:59
INFO     2024-02-06 01:06:17,646 sentence_transformers.SentenceTransformer INFO: Load pretrained SentenceTransformer: /mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125                                                                                               SentenceTransformer.py:110
/app/.venv/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
ERROR:    Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/util.py", line 533, in load_dir_path
    repo_path = snapshot_download(**download_kwargs)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125'. Use `repo_type` argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 654, in startup
    await handler()
  File "/app/infinity_emb/infinity_server.py", line 70, in _startup
    app.model = AsyncEmbeddingEngine(
  File "/app/infinity_emb/engine.py", line 74, in __init__
    self._model, self._min_inference_t, self._max_inference_t = select_model(
  File "/app/infinity_emb/inference/select_model.py", line 66, in select_model
    loaded_engine = unloaded_engine.value(
  File "/app/infinity_emb/transformer/embedder/sentence_transformer.py", line 49, in __init__
    super().__init__(
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 194, in __init__
    modules = self._load_sbert_model(
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1062, in _load_sbert_model
    module_path = load_dir_path(
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/util.py", line 537, in load_dir_path
    repo_path = snapshot_download(**download_kwargs)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125'. Use `repo_type` argument if needed.

ERROR:    Application startup failed. Exiting.

For some reason it picks the following path SentenceTransformer#L1062

Where it joins 2 paths, resulting in a path that doesn't exist, after which it calls snapshot_download which fails.

What am I doing wrong? I should be able to run this backend in offline mode right?

[michaelfeil]

Hey @gerritvd, thanks for your question. For now I would recommend not using a local_path as the support for this seems complicated with sentence-transformers 2.3.0.

I would resort to one of the following: Use either 0.0.20 as pip or docker image.
https://github.com/michaelfeil/infinity/releases/tag/0.0.20
Or prime the cache by downloading to the default huggingface snapshot_download folder, without specifiying a extra dir.

[gerritvd]

Thanks for responding so quickly. v0.0.20 did indeed work. What changed between ST implementations?
The reason for wanting to specify a path is that we have a different model download than from HF.

[michaelfeil]

Gotcha, would you be so kind and test if it runs with sentence-transformers 2.3.1 directly (without infinity). If it does not, and it does with 2.2.2, please open an issue on sentence-transformers? 😊

[gerritvd]

SentenceTransformers 2.3.1 does work with a path.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer(f"/home/{user}/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125/", device='cpu')

works fine and uses the path to load the model.

I tried running a minimal version of infinity 0.0.23 the way it works for me with 0.0.20 but get a different error:
This works with 0.0.20:

docker run -e HF_HUB_OFFLINE=0 -e SENTENCE_TRANSFORMERS_HOME=/mnt/models/hub -v/home/$USER/.cache/huggingface/hub:/mnt/models/hub -p7997:7997 -it --rm --entrypoint=infinity_emb infinity:v0.0.20 --model-name-or-path=sentence-transformers/all-MiniLM-L6-v2

Now the data is cached (but assume I want to prime without hf hub). Now I can run it using:

docker run -e HF_HUB_OFFLINE=1 -v/home/$USER/.cache/huggingface/hub:/mnt/models/hub -p7997:7997 -it --rm --entrypoint=infinity_emb infinity:v0.0.20 --model-name-or-path=/mnt/models/hub/sentence-transformers_all-MiniLM-L6-v2

This works with v0.0.20. However, when I used the exact same commands with 0.0.23 I need to change the path because it stores it in:

~/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125

vs in 0.0.20 in:

~/.cache/huggingface/hub/sentence-transformers_all-MiniLM-L6-v2

Now when I run it in offline mode:

docker run -e HF_HUB_OFFLINE=1 -v/home/$USER/.cache/huggingface/hub:/mnt/models/hub -p7997:7997 -it --rm --entrypoint=infinity_emb infinity:v0.0.23 --model-name-or-path=/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125

I get the error:

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO     2024-02-08 04:02:37,282 infinity_emb INFO: model=`/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125` selected, using engine=`torch` and device=`None`                                                                                                                select_model.py:59
INFO     2024-02-08 04:02:37,285 sentence_transformers.SentenceTransformer INFO: Load pretrained SentenceTransformer: /mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125                                                                                               SentenceTransformer.py:110
/app/.venv/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
ERROR:    Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/util.py", line 533, in load_dir_path
    repo_path = snapshot_download(**download_kwargs)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125'. Use `repo_type` argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 654, in startup
    await handler()
  File "/app/infinity_emb/infinity_server.py", line 70, in _startup
    app.model = AsyncEmbeddingEngine(
  File "/app/infinity_emb/engine.py", line 74, in __init__
    self._model, self._min_inference_t, self._max_inference_t = select_model(
  File "/app/infinity_emb/inference/select_model.py", line 66, in select_model
    loaded_engine = unloaded_engine.value(
  File "/app/infinity_emb/transformer/embedder/sentence_transformer.py", line 49, in __init__
    super().__init__(
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 194, in __init__
    modules = self._load_sbert_model(
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1062, in _load_sbert_model
    module_path = load_dir_path(
  File "/app/.venv/lib/python3.10/site-packages/sentence_transformers/util.py", line 537, in load_dir_path
    repo_path = snapshot_download(**download_kwargs)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "/app/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/mnt/models/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125'. Use `repo_type` argument if needed.

Note that running 0.0.20 with the 0.0.23 path also works fine.

[michaelfeil]

Thanks for the detailed description.

I suspect that huggingface_hub, works fine .e.g when you are having the model in the place where export HF_HOME is expecting it.

If you are using infinity outside of docker e.g. pip install infinity_emb && `infinity_emb --model-name-or-path "/home/{user}/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/7dbbc90392e2f80f3d3c277d6e90027e55de9125/"

[michaelfeil]

Likley also SENTENCE_TRANSFORMERS_HOME is no longer respected starting from sentence-transformers 0.0.23

[michaelfeil]

To get creative, you can also build a docker image starting from FROM michaelf34/infinity, execute RUN infinity_emb --model-name-or-path once, and then use the normal entrypoint.