Switch to Pixi for development / CI and hatchling for build system #6182

hoxbro · 2024-04-10T16:49:40Z

Just for playing around with it.

Task issue: holoviz-dev/holoviz_tasks#34

TODO (non-exhaustive)

Figure out how tsdownsample can only be added to certain arch
Add developer documentation (http://dev.holoviews.org/developer_guide/index.html)
Remove the hatch_build.py script, as I was trying things out. (Is needed for Panel)
Add comments and describe what has been done in this PR.
If we should use pixi.toml or put it into pyproject.toml (no strong opinions)
- For now, let's keep them separated.
Figure out how to use editable as it currently solves each time you run pixi run or pixi shell
- I have decided to have an install command. Which needs to be run if you like to editable install. It seems like pixi will likely support triggers like post-install which makes sense for this.
Why sdist is needed when building conda build - is because of Unable to install from sdist using force-include with hatchling>=1.19 pypa/hatch#1305.

A rough sketch of how I see things (aka. not something set in stone)

Current output of pixi info:

❯ pixi info
      Pixi version: 0.19.1
          Platform: linux-64
  Virtual packages: __unix=0=0
                  : __linux=6.8.0=0
                  : __glibc=2.35=0
                  : __cuda=12.4=0
                  : __archspec=1=zen3
         Cache dir: /home/shh/.cache/rattler/cache
      Auth storage: /home/shh/.rattler/credentials.json

Project
------------
     Manifest file: /home/shh/projects/holoviz/repos/holoviews/pixi.toml
  Config locations: /home/shh/.config/pixi/config.toml, /home/shh/.config/pixi/config.toml
      Last updated: 18-04-2024 18:43:38

Environments
------------
       Environment: default
          Features: default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 13
      Dependencies: python, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, notebook, matplotlib-base, bokeh
  Target platforms: win-64, linux-64, osx-arm64, osx-64
             Tasks: check-latest-packages, download-data, install

       Environment: test-39
          Features: py39, test-core, test, example, test-example, test-unit-task, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 41
      Dependencies: python, pytest, pytest-cov, pytest-github-actions-annotate-failures, pytest-rerunfailures, nbconvert, pillow, plotly, contourpy, dask-core, ibis-sqlite, xarray, networkx, shapely, ffmpeg, cftime, scipy, selenium, spatialpandas, datashader, xyzservices, tsdownsample, streamz, netcdf4, scikit-image, pyarrow, pooch, notebook, nbval, pytest-xdist, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, matplotlib-base, bokeh
  Target platforms: osx-64, win-64, linux-64, osx-arm64
             Tasks: download-data, check-latest-packages, test-unit, install, test-example

       Environment: test-310
          Features: py310, test-core, test, example, test-example, test-unit-task, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 41
      Dependencies: python, pytest, pytest-cov, pytest-github-actions-annotate-failures, pytest-rerunfailures, nbconvert, pillow, plotly, contourpy, dask-core, ibis-sqlite, xarray, networkx, shapely, ffmpeg, cftime, scipy, selenium, spatialpandas, datashader, xyzservices, tsdownsample, streamz, netcdf4, scikit-image, pyarrow, pooch, notebook, nbval, pytest-xdist, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, matplotlib-base, bokeh
  Target platforms: osx-64, win-64, linux-64, osx-arm64
             Tasks: check-latest-packages, test-unit, test-example, download-data, install

       Environment: test-311
          Features: py311, test-core, test, example, test-example, test-unit-task, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 41
      Dependencies: python, pytest, pytest-cov, pytest-github-actions-annotate-failures, pytest-rerunfailures, nbconvert, pillow, plotly, contourpy, dask-core, ibis-sqlite, xarray, networkx, shapely, ffmpeg, cftime, scipy, selenium, spatialpandas, datashader, xyzservices, tsdownsample, streamz, netcdf4, scikit-image, pyarrow, pooch, notebook, nbval, pytest-xdist, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, matplotlib-base, bokeh
  Target platforms: osx-arm64, linux-64, osx-64, win-64
             Tasks: check-latest-packages, test-example, test-unit, install, download-data

       Environment: test-312
          Features: py312, test-core, test, example, test-example, test-unit-task, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 41
      Dependencies: python, pytest, pytest-cov, pytest-github-actions-annotate-failures, pytest-rerunfailures, nbconvert, pillow, plotly, contourpy, dask-core, ibis-sqlite, xarray, networkx, shapely, ffmpeg, cftime, scipy, selenium, spatialpandas, datashader, xyzservices, tsdownsample, streamz, netcdf4, scikit-image, pyarrow, pooch, notebook, nbval, pytest-xdist, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, matplotlib-base, bokeh
  Target platforms: osx-64, win-64, osx-arm64, linux-64
             Tasks: install, download-data, check-latest-packages, test-example, test-unit

       Environment: test-ui
          Features: py312, test-core, test, test-ui, default
          Channels: conda-forge, pyviz/label/dev, microsoft
  Dependency count: 36
      Dependencies: python, pytest, pytest-cov, pytest-github-actions-annotate-failures, pytest-rerunfailures, nbconvert, pillow, plotly, contourpy, dask-core, ibis-sqlite, xarray, networkx, shapely, ffmpeg, cftime, scipy, selenium, spatialpandas, datashader, xyzservices, tsdownsample, playwright, pytest-playwright, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, notebook, matplotlib-base, bokeh
  Target platforms: osx-arm64, osx-64, win-64, linux-64
             Tasks: check-latest-packages, test-ui, download-data, install-ui, install

       Environment: test-core
          Features: py312, test-core, test-unit-task, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 21
      Dependencies: python, pytest, pytest-cov, pytest-github-actions-annotate-failures, pytest-rerunfailures, nbconvert, pillow, plotly, contourpy, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, notebook, matplotlib-base, bokeh
  Target platforms: win-64, osx-arm64, linux-64, osx-64
             Tasks: download-data, check-latest-packages, install, test-unit

       Environment: docs
          Features: py311, example, doc, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 31
      Dependencies: python, networkx, pillow, xarray, plotly, streamz, ffmpeg, cftime, netcdf4, dask-core, scipy, shapely, scikit-image, pyarrow, pooch, datashader, notebook, nbsite, graphviz, selenium, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, matplotlib-base, bokeh
  Target platforms: osx-64, linux-64, osx-arm64, win-64
             Tasks: docs-build, check-latest-packages, install, docs-refmanual, download-data, docs-generate-rst, docs-generate

       Environment: build
          Features: py311, build, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 15
      Dependencies: python, build, conda-build, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, notebook, matplotlib-base, bokeh
  Target platforms: osx-arm64, linux-64, win-64, osx-64
             Tasks: build-conda, check-latest-packages, install, download-data, build-pip

       Environment: lint
          Features: py311, lint, default
          Channels: conda-forge, pyviz/label/dev
  Dependency count: 14
      Dependencies: python, pre-commit, pip, param, panel, pyviz_comms, colorcet, numpy, packaging, pandas, ipython, notebook, matplotlib-base, bokeh
  Target platforms: osx-arm64, linux-64, osx-64, win-64
             Tasks: install, lint, check-latest-packages, download-data, lint-install

hoxbro · 2024-04-18T07:29:24Z

Hatchling in pyproject.toml:

[tool.hatch.build.hooks.custom]
path = "scripts/hatch_build.py"

The hatchling script

from __future__ import annotations

import typing as t

from hatchling.builders.hooks.plugin.interface import BuildHookInterface


class BuildHook(BuildHookInterface):
    """The hatch jupyter builder build hook."""

    PLUGIN_NAME = "install"

    def initialize(self, version: str, _: dict[str, t.Any]) -> None:
        """Initialize the plugin."""
        if self.target_name not in ["wheel", "sdist"]:
            return

        if version == "editable":
            ...

hoxbro · 2024-04-18T07:40:25Z

.github/workflows/test.yaml

-          doit test_examples
-      - name: codecov
+          pixi run -e ${{ matrix.environment }} test-example --cov=./holoviews --cov-report=xml --cov-append
+      - uses: codecov/codecov-action@v4


This is not strictly related to the Pixi, but I noticed that it was failing. So I updated it to use the official workflow.

.gitignore

hoxbro · 2024-04-18T07:48:37Z

holoviews/tests/util/test_utils.py

@@ -26,6 +26,8 @@ class TestOutputUtil(ComparisonTestCase):
    def setUp(self):
        if notebook is None:
            raise SkipTest("Jupyter Notebook not available")
+        from holoviews.ipython import notebook_extension


It seems like IPython is not part of the sys.module at the import time, but I'm not entirely sure why.

hoxbro · 2024-04-18T07:50:03Z

scripts/conda/build.sh

+
+for file in dist/*.whl dist/*.tar.bz2; do
+    if [ -e "$file" ]; then
+        echo "dist folder already contains $(basename "$file"). Please delete it before running this script."


If not, this will give some obscure error when running conda-build.

hoxbro · 2024-04-18T07:50:18Z

scripts/conda/build.sh

+export VERSION
+conda build scripts/conda/recipe --no-anaconda-upload --no-verify
+
+mv "$CONDA_PREFIX/conda-bld/noarch/$PACKAGE-$VERSION-py_0.tar.bz2" dist


To make it easier to get the file.

hoxbro · 2024-04-18T07:50:51Z

scripts/download_data.py

@@ -0,0 +1,13 @@
+import bokeh.sampledata


Changed from bash to python because there was no need for it to be a bash file...

time

hoxbro · 2024-04-18T17:28:09Z

.github/workflows/test.yaml

-  OMP_NUM_THREADS: 1
-  OPENBLAS_NUM_THREADS: 1
-  MKL_NUM_THREADS: 1
-  VECLIB_MAXIMUM_THREADS: 1
-  NUMEXPR_NUM_THREADS: 1
-  NUMBA_NUM_THREADS: 1
-  PYDEVD_DISABLE_FILE_VALIDATION: 1


I have moved those to the install task.

hoxbro · 2024-04-19T06:27:07Z

.github/workflows/test.yaml

@@ -90,153 +83,111 @@ jobs:
        if: env.MATRIX_OPTION == 'default'
        run: |
          MATRIX=$(jq -nsc '{
-              "os": ["ubuntu-latest", "macos-latest", "windows-latest"],
-              "python-version": ["3.9", "3.12"]
+              "os": ["ubuntu-latest", "macos-14", "windows-latest"],


I changed to using Mac-arm64 as this is both faster and the newer architecture.

github-actions · 2024-10-23T10:28:47Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

hoxbro added 4 commits April 10, 2024 15:47

Move deps to pyproject.toml

79b3794

Update other setup.py stuff

2ec0c05

Try to move stuff to pixi

a7933b2

Try updating the ci test file

d760350

hoxbro force-pushed the pixi_play branch 2 times, most recently from b40b746 to f5d0e56 Compare April 10, 2024 17:00

Small changes

ce6037b

hoxbro force-pushed the pixi_play branch from f5d0e56 to ce6037b Compare April 10, 2024 17:03

Remove setup.py

726ed7e

hoxbro added the type: maintenance infrastructure and maintenance of CI label Apr 10, 2024

Add global tasks

0062cb1

hoxbro force-pushed the pixi_play branch from a65c4cd to 17b7c7d Compare April 10, 2024 17:33

Add environment

6a52141

hoxbro force-pushed the pixi_play branch from 17b7c7d to 6a52141 Compare April 10, 2024 17:35

hoxbro added 6 commits April 10, 2024 19:54

Add ui and core tests

e15620e

Updates

93cec10

Remove setup.cfg

6917525

Rename python-version to environment

c5b93ce

Add PIXI_ENVIRONMENT_NAME and install-ui

a5b2172

Update environments

d179b11

hoxbro force-pushed the pixi_play branch from 9114811 to d179b11 Compare April 11, 2024 07:36

hoxbro added 3 commits April 11, 2024 09:50

Revert PIXI_ENVIRONMENT_NAME

da8b2c0

Clean up

f492143

Add pixi.lock step

fc62bd5

hoxbro force-pushed the pixi_play branch from 34b8372 to fc62bd5 Compare April 11, 2024 08:32

hoxbro added 4 commits April 11, 2024 10:53

examples -> example

aba4511

Disable dask_expr for now

bb0eeb0

Update UI tests to work with osx-arm64

cfee98e

Use depends on for test-ui

46a9a9a

hoxbro force-pushed the pixi_play branch from ac65d49 to 46a9a9a Compare April 11, 2024 10:16

hoxbro added 2 commits April 16, 2024 14:23

Add xyzservices

ea77fe4

Adding coverage to examples

37ccc9b

hoxbro force-pushed the pixi_play branch from 822d847 to 37ccc9b Compare April 18, 2024 07:08

hoxbro added 2 commits April 18, 2024 09:29

Remove hatchling setupscript

3a7a6ac

Ignore __version.py file from coverage

308eb51

hoxbro commented Apr 18, 2024

View reviewed changes

hoxbro added 11 commits April 18, 2024 10:46

Add lint environment

5a2145c

Remove test-unit for the test-ui environment

d2891e8

Add developer guide

fb90b42

Remove environment variables as they are directly in the action

f4a18e4

Exclude lines from covarage

f92a285

Remove coverage from examples as it significantly slows down the run

ad08c33

time

Update developer guide

8c74dfe

Add to index.rst

7184033

Update extend-ignore

788cf02

Docs update

ab113a6

Update documentation dev guide

9867caf

hoxbro commented Apr 18, 2024

View reviewed changes

hoxbro marked this pull request as ready for review April 18, 2024 17:29

Disable dask-expr for docs build

8bd7937

hoxbro commented Apr 19, 2024

View reviewed changes

philippjfr approved these changes Apr 19, 2024

View reviewed changes

hoxbro changed the title ~~Try pixi out~~ Switch to Pixi for development / CI and hatchling for build system Apr 19, 2024

hoxbro merged commit 4727470 into main Apr 19, 2024
14 of 15 checks passed

hoxbro deleted the pixi_play branch April 19, 2024 15:29

This was referenced Apr 20, 2024

Update documentation and small Pixi improvements #6200

Merged

Switch to Pixi for development / CI and hatchling for build system holoviz/panel#6775

Merged

github-actions bot locked as resolved and limited conversation to collaborators Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to Pixi for development / CI and hatchling for build system #6182

Switch to Pixi for development / CI and hatchling for build system #6182

hoxbro commented Apr 10, 2024 •

edited

Loading

hoxbro commented Apr 18, 2024

hoxbro Apr 18, 2024

hoxbro Apr 18, 2024 •

edited

Loading

hoxbro Apr 18, 2024

hoxbro Apr 18, 2024

hoxbro Apr 18, 2024

hoxbro Apr 18, 2024

hoxbro Apr 19, 2024 •

edited

Loading

github-actions bot commented Oct 23, 2024

Switch to Pixi for development / CI and hatchling for build system #6182

Switch to Pixi for development / CI and hatchling for build system #6182

Conversation

hoxbro commented Apr 10, 2024 • edited Loading

TODO (non-exhaustive)

hoxbro commented Apr 18, 2024

hoxbro Apr 18, 2024

Choose a reason for hiding this comment

hoxbro Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

hoxbro Apr 18, 2024

Choose a reason for hiding this comment

hoxbro Apr 18, 2024

Choose a reason for hiding this comment

hoxbro Apr 18, 2024

Choose a reason for hiding this comment

hoxbro Apr 18, 2024

Choose a reason for hiding this comment

hoxbro Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Oct 23, 2024

hoxbro commented Apr 10, 2024 •

edited

Loading

hoxbro Apr 18, 2024 •

edited

Loading

hoxbro Apr 19, 2024 •

edited

Loading