build(nix): Package gguf-py #5664

ditsuke · 2024-02-22T15:37:01Z

What and why

Currently, the Nix build does not ship with the gguf-py lib, rendering scripts like convert-hf-to-gguf unusable.

This is an attempt to package gguf-py with buildPythonPackage. Very barebones at the moment and needs some work to exclude the Nix built gguf from the devShells, and some thought into other things like packaging the heavier python dependencies (we don't want them to be included by default, but we need a way to opt-in).

CC: @SomeoneSerge

.devops/nix/package.nix

ditsuke · 2024-02-23T12:20:35Z

This PR is good for review for the standalone packaging of gguf-py. However for the convert* scripts that ship with this repo to work, they need a Python environment that has the full superset of dependencies they rely on including torch. We'll need to ship with these heavy dependencies, or alternatively we could:

Decouple the scripts from default llama package, and expose them through a llama-scripts package instead.
By default have them run against the default python interpreter in the environment and ship the static-environment version as another option.

Open to other ideas.

SomeoneSerge · 2024-02-23T14:16:42Z

.devops/nix/package-gguf-py.nix

+{
+  lib,
+  llamaVersion,
+  python3,


buildPythonPackage, poetry-core, numpy as arguments. No need for python3

The evaluating scope should set them explicitly then?

Not sure if there's a cleaner way: 00fbda0

No, the scope inherits from python3Packages, which in turn inherits from pkgs

Ah, sorry, mistake on my part. This one only inherits from pkgs, we need to handle python separately.

EDIT: this is good, we'll see about making one for python, and making it easy to reuse in pythonPackagesExtensions

Okay, then we're good to go for this PR?

If by "good to go" you mean we need to move the python scripts out of the main derivation so that it doesn't depend on the choice of python version 😅

EDIT: I phrased this badly. What I meant is, I think the way to introduce support for the python scripts is to separate the python and C++ parts and to add means to extend python3Packages. I'm pretty exhausted today and won't be able to make my explanation any better, but I'll try to send some code tomorrow. I do acknowledge that we shouldn't extend the scope of the PR too much and you likely want this feature merged in a timely manner, etc, etc, but right now I think there's a bit more work to be done

No problem. I gave splitting the Python/C++ devShells and derivation a shot. I don't think it addresses extending python3Packages, but it should address the other concern about the mixed-in Python and C++ derivation, and get the python scripts exposed with .#python-scripts. Please take a look and tell me what you think, then maybe we can get the python3Packages thing done and get this in :D

.devops/nix/package.nix

.devops/nix/package-gguf-py.nix

.devops/nix/package.nix

bin/convert_hf_to_gguf.py

pyproject.toml

.devops/nix/nixpkgs-instances.nix

ditsuke · 2024-07-08T12:00:00Z

Rebased to after merge of #5745. Commits will need to be cleaned up, but this is open for review otherwise.

.devops/nix/python-scripts.nix

.devops/nix/package.nix

compilade

This seems to work for my workflow with nix develop .#default-extra. Thanks!

I've tested running pyright on the whole repository (which requires access to all python libraries used within), and it works without problem.

I've also tested building a -DGGML_VULKAN=TRUE build with nix develop .#vulkan-extra, and it works too.

There are more things in the $PATH (because python libraries only were added to the python interpreter before, not also the $PATH), but I think this might be useful in some situations.

Rebuilding gguf-py at each change might be cumbersome, though, but I think this is better than before because I simply never tested building gguf-py with poetry before, but with this it's now handled when creating a devShell. (I never needed to actually build the package because convert_hf_to_gguf.py uses some other trick anyway to include the local gguf-py instead of the package)

I think I like this.

.devops/nix/package.nix

.devops/nix/python-scripts.nix

Nix has an older version

- Filter out non-source files from llama-scripts flake derivation - Clean up unused closure - Remove scripts devShell

ditsuke · 2024-09-02T10:01:10Z

Rebased to master. @ggerganov this should be ready for merge. cc @SomeoneSerge

ditsuke · 2024-09-02T13:13:59Z

Thanks for merging!

SomeoneSerge · 2024-09-02T13:53:51Z

.devops/nix/scope.nix

  llamaVersion ? "0.0.0",
 }:

+let
+  pythonPackages = python3.pkgs;


A note for the future: I'd discourage using python3.pkgs in favour of python3Packages because the former does not support splicing. We're not using any cross-compilation in this flake, but I think it's important to stay minimal and idiomatic so as not to spread confusion

That's good to know. This should be changed to use python3Packages next time someone visits these files.

SomeoneSerge · 2024-09-02T13:54:38Z

.devops/nix/scope.nix

+  buildPythonPackage = pythonPackages.buildPythonPackage;
+  numpy = pythonPackages.numpy;
+  tqdm = pythonPackages.tqdm;
+  sentencepiece = pythonPackages.sentencepiece;
+  pyyaml = pythonPackages.pyyaml;
+  poetry-core = pythonPackages.poetry-core;
+  pytestCheckHook = pythonPackages.pytestCheckHook;


None of this would be necessary if we made a subscope for python packages, but this can be left for future. And the UX for implementing that currently sucks anyway

Yes would be sure nice to have a python scope at some point, this is rather ugly.

* style: format with nixfmt/rfc101-style * build(nix): Package gguf-py * build(nix): Refactor to new scope for gguf-py * build(nix): Exclude gguf-py from devShells * build(nix): Refactor gguf-py derivation to take in exact deps * build(nix): Enable pytestCheckHook and pythonImportsCheck for gguf-py * build(python): Package python scripts with pyproject.toml * chore: Cleanup * dev(nix): Break up python/C devShells * build(python): Relax pytorch version constraint Nix has an older version * chore: Move cmake to nativeBuildInputs for devShell * fmt: Reconcile formatting with rebase * style: nix fmt * cleanup: Remove unncessary __init__.py * chore: Suggestions from review - Filter out non-source files from llama-scripts flake derivation - Clean up unused closure - Remove scripts devShell * revert: Bad changes * dev: Simplify devShells, restore the -extra devShell * build(nix): Add pyyaml for gguf-py * chore: Remove some unused bindings * dev: Add tiktoken to -extra devShells

SomeoneSerge reviewed Feb 22, 2024

View reviewed changes

.devops/nix/package.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Feb 22, 2024

View reviewed changes

.devops/nix/package.nix Outdated Show resolved Hide resolved

ditsuke force-pushed the build/nix/gguf-py-inc branch 2 times, most recently from 7efb80d to 1d0a97c Compare February 23, 2024 09:55

ditsuke marked this pull request as ready for review February 23, 2024 12:01

SomeoneSerge reviewed Feb 23, 2024

View reviewed changes

.devops/nix/package.nix Show resolved Hide resolved

ditsuke force-pushed the build/nix/gguf-py-inc branch 2 times, most recently from 4957125 to 00fbda0 Compare February 23, 2024 17:32

SomeoneSerge reviewed Feb 24, 2024

View reviewed changes

.devops/nix/package-gguf-py.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Feb 24, 2024

View reviewed changes

.devops/nix/package.nix Outdated Show resolved Hide resolved

ditsuke force-pushed the build/nix/gguf-py-inc branch from 28d0c6d to ad57699 Compare February 26, 2024 19:00

SomeoneSerge reviewed Feb 26, 2024

View reviewed changes

.devops/nix/package.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Feb 26, 2024

View reviewed changes

bin/convert_hf_to_gguf.py Outdated Show resolved Hide resolved

SomeoneSerge reviewed Feb 26, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

ditsuke mentioned this pull request Feb 27, 2024

build(python): Package scripts with pep-0517 compliance #5745

Merged

SomeoneSerge reviewed Mar 1, 2024

View reviewed changes

.devops/nix/nixpkgs-instances.nix Outdated Show resolved Hide resolved

ditsuke force-pushed the build/nix/gguf-py-inc branch from 5ab8e9c to cbd8200 Compare March 10, 2024 16:28

SomeoneSerge added the nix Issues specific to consuming flake.nix, or generally concerned with ❄ Nix-based llama.cpp deployment label Mar 21, 2024

ditsuke force-pushed the build/nix/gguf-py-inc branch from cbd8200 to 0a82d07 Compare July 8, 2024 11:57

SomeoneSerge reviewed Jul 8, 2024

View reviewed changes

.devops/nix/python-scripts.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Jul 8, 2024

View reviewed changes

.devops/nix/python-scripts.nix Outdated Show resolved Hide resolved

github-actions bot added python python script changes devops improvements to build systems and github actions labels Jul 8, 2024

ditsuke force-pushed the build/nix/gguf-py-inc branch from 54ce50d to 5493dd8 Compare July 8, 2024 18:11

compilade reviewed Jul 8, 2024

View reviewed changes

.devops/nix/package.nix Show resolved Hide resolved

ditsuke force-pushed the build/nix/gguf-py-inc branch 2 times, most recently from 7729067 to 0188318 Compare August 19, 2024 21:28

compilade approved these changes Aug 30, 2024

View reviewed changes

.devops/nix/package.nix Outdated Show resolved Hide resolved

.devops/nix/python-scripts.nix Show resolved Hide resolved

ditsuke added 20 commits September 2, 2024 15:29

style: format with nixfmt/rfc101-style

0126788

build(nix): Package gguf-py

c3bc2f6

build(nix): Refactor to new scope for gguf-py

f363d30

build(nix): Exclude gguf-py from devShells

0b8ddf8

build(nix): Refactor gguf-py derivation to take in exact deps

11e581b

build(nix): Enable pytestCheckHook and pythonImportsCheck for gguf-py

4b124fb

build(python): Package python scripts with pyproject.toml

10a49b9

chore: Cleanup

51056d9

dev(nix): Break up python/C devShells

ef2dae9

build(python): Relax pytorch version constraint

d7b5776

Nix has an older version

chore: Move cmake to nativeBuildInputs for devShell

a62a2c2

fmt: Reconcile formatting with rebase

b7611bf

style: nix fmt

8e911d7

cleanup: Remove unncessary __init__.py

4a71d86

chore: Suggestions from review

03b2afe

- Filter out non-source files from llama-scripts flake derivation - Clean up unused closure - Remove scripts devShell

revert: Bad changes

cd8da29

dev: Simplify devShells, restore the -extra devShell

06a8547

build(nix): Add pyyaml for gguf-py

dcee475

chore: Remove some unused bindings

fa0e7c9

dev: Add tiktoken to -extra devShells

bd574ba

ditsuke force-pushed the build/nix/gguf-py-inc branch from e61d467 to bd574ba Compare September 2, 2024 10:00

ggerganov merged commit 9c1ba55 into ggerganov:master Sep 2, 2024
8 checks passed

SomeoneSerge reviewed Sep 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build(nix): Package gguf-py #5664

build(nix): Package gguf-py #5664

ditsuke commented Feb 22, 2024 •

edited

Loading

ditsuke commented Feb 23, 2024

SomeoneSerge Feb 23, 2024 •

edited

Loading

ditsuke Feb 23, 2024

ditsuke Feb 23, 2024 •

edited

Loading

SomeoneSerge Feb 23, 2024

SomeoneSerge Feb 23, 2024 •

edited

Loading

ditsuke Feb 24, 2024

SomeoneSerge Feb 24, 2024 •

edited

Loading

ditsuke Feb 26, 2024

ditsuke commented Jul 8, 2024

compilade left a comment

ditsuke commented Sep 2, 2024

ditsuke commented Sep 2, 2024

SomeoneSerge Sep 2, 2024

ditsuke Sep 2, 2024

SomeoneSerge Sep 2, 2024

ditsuke Sep 2, 2024

build(nix): Package gguf-py #5664

build(nix): Package gguf-py #5664

Conversation

ditsuke commented Feb 22, 2024 • edited Loading

What and why

ditsuke commented Feb 23, 2024

SomeoneSerge Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ditsuke Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SomeoneSerge Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SomeoneSerge Feb 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ditsuke commented Jul 8, 2024

compilade left a comment

Choose a reason for hiding this comment

ditsuke commented Sep 2, 2024

ditsuke commented Sep 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ditsuke commented Feb 22, 2024 •

edited

Loading

SomeoneSerge Feb 23, 2024 •

edited

Loading

ditsuke Feb 23, 2024 •

edited

Loading

SomeoneSerge Feb 23, 2024 •

edited

Loading

SomeoneSerge Feb 24, 2024 •

edited

Loading