How to override poetry python packages using JupyterWith UI?

[michelav]

I need to extend kernel by using JupyterWith's configured poetry (can't build some libs when using extraPackages #395 ). I've tried to define my dependencies in pyproject.toml but it still fails for some libs (e.g. scikit-learn). There is a known issue related to that at poetry2nix. Meanwhile there is no better solution, I guess I'd have to override some libs (in my case packaging) and include fit-core as buildInput.

I'm trying to do that through an overrides file in the same dir my custom kernel is defined.

├── default.nix
├── overrides.nix
├── poetry.lock
└── pyproject.toml

This is the customized kernel:

{ name, availableKernels, extraArgs, ... }:
availableKernels.python {
  projectDir = ./.;
  displayName = "PD4ML Python";
  inherit (extraArgs) system;
  name = "pd4ml-python";
}

and the overrides file.

final: prev: {
  packaging = prev.packaging.overridePythonAttrs (old: {
    buildInputs = old.buildInputs or [ ] ++ [ final.python3.pkgs.flit-core ];
  });
}

This is my pyproject.toml.

[tool.poetry]
name = "pd4ml-python"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.9"
ipykernel = "*"
scikit-learn = "*"

[tool.poetry.dev-dependencies]
# build systems for dependencies
hatchling = "^1.3.1"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

After that I keep getting ModuleNotFoundError: No module named 'flit_core'.

Is this the right way to override packages in poetry through JupyterWith?

[michelav]

OK. Got it working. To extend the kernel using poetry one must set up the projectDir so all setup files are read from there. However, the overrides attribute isn't set accordingly.

This was extracted from default python kernel. I guess overrides attribute should be read from projectDir as well.

{
 projectDir ? self + "/kernels/available/python",
  pyproject ? projectDir + "/pyproject.toml",
  poetrylock ? projectDir + "/poetry.lock",
  overrides ? poetry2nix.overrides.withDefaults (import ./overrides.nix),
}

In my case I had to add poetry2nix as input, set it up and plug it as extraArgs when configuring my JupyterLab.

Flake.nix

let
        python = pkgs.python3;
        poetry2nix =
          import "${inputs.poetry2nix}/default.nix" { inherit pkgs poetry; };

        poetry = pkgs.callPackage "${inputs.poetry2nix}/pkgs/poetry" {
          inherit python;
        };
        inherit (jupyterWith.lib.${system}) mkJupyterlabFromPath;
        jupyterlab =
          mkJupyterlabFromPath ./kernels { inherit system poetry2nix poetry; };
 in { ... }

kernels/custom-python/default.nix

{ name, availableKernels, extraArgs, ... }:
availableKernels.python rec {
  projectDir = ./.;
  displayName = "PD4ML Python";
  inherit (extraArgs) system poetry2nix;
  overrides = poetry2nix.overrides.withDefaults (final: prev: {
    packaging = prev.packaging.overridePythonAttrs
      (old: { buildInputs = old.buildInputs or [ ] ++ [ prev.flit-core ]; });
  });
  name = "pd4ml-python";
}

[djacu]

Glad you got it working!