Failed building wheel for safetensors

[HamedMP]

Hi,

Super interesting project.

I created a new conda env with python 11 conda create -n khoj python=3.11 and ran the pip installation, but got the following error.

Any idea how to resolve it?

I'm on Mac M1, with the latest MacOS 13.5.1.

      Some errors have detailed explanations: E0405, E0412, E0425, E0433, E0463, E0531.
      For more information about an error, try `rustc --explain E0405`.
      error: could not compile `smallvec` due to 145 previous errors

      Caused by:
        process didn't exit successfully: `rustc --crate-name smallvec --edition=2018 /Users/hamed/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=b41f1767cebac6a1 -C extra-filename=-b41f1767cebac6a1 --out-dir /private/var/folders/zl/m2lw7vkd495djcf9x2499md80000gn/T/pip-install-zmbcfh5z/safetensors_22bdd82a17b14939a5597ba399186ebe/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/private/var/folders/zl/m2lw7vkd495djcf9x2499md80000gn/T/pip-install-zmbcfh5z/safetensors_22bdd82a17b14939a5597ba399186ebe/target/x86_64-apple-darwin/release/deps -L dependency=/private/var/folders/zl/m2lw7vkd495djcf9x2499md80000gn/T/pip-install-zmbcfh5z/safetensors_22bdd82a17b14939a5597ba399186ebe/target/release/deps --cap-lints allow` (exit status: 1)
      warning: build failed, waiting for other jobs to finish...
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --target x86_64-apple-darwin --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/_safetensors_rust.cpython-311-darwin.so'` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for safetensors
Failed to build safetensors
ERROR: Could not build wheels for safetensors, which is required to install pyproject.toml-based projects

[sabaimran]

Hey @HamedMP ! Sorry about the setup issues. i've seen this before, and the root cause is usually a missing rust installation.

these instructions would be helpful for installing rust -- https://docs.khoj.dev/#/setup?id=install-fails-while-building-tokenizer-dependency. Could you try that out and let me know if it works?

brew install rustup
rustup-init
source ~/.cargo/env

[HamedMP]

Thanks @sabaimran, I've tried rustup-init but I get the error that PATH already exist.

error: could not create link from '/Users/hamed/.cargo/bin/rustup' to '/Users/hamed/.cargo/bin/rustfmt': File exists (os error 17)

Running rustc --version gives:

❯ rustc --version
rustc 1.66.0 (69f9c33d7 2022-12-12)

[HamedMP]

I just looked up safetensors specifically and found this thread

Installing rust in my conda environment resolved the issue. So it was mainly conda-related.

❯ conda install -c conda-forge rust                                       

P.s. After running khoj, I didn't see any info that I need to go to http://127.0.0.1:42110 to see the interface. Adding it to the doc would be amazing <3

Found the URL from the Obsidian plugin settings 😄