Fix `docs-check` job; numpy 2.x fixes #157

sk1p · 2024-06-11T10:30:10Z

Try to be consistent with other jobs.

Contributor Checklist:

I have added or updated my entry in the creators.json file
I have added a changelog entry for my contribution
I have added/updated documentation for all user-facing changes
I have added/updated test cases

Reviewer Checklist:

/azp run libertem.libertem-live-data passed

sk1p · 2024-06-11T10:32:39Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-11T10:32:48Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov · 2024-06-11T10:33:08Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.84%. Comparing base (3562273) to head (acae5ff).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #157   +/-   ##
=======================================
  Coverage   79.83%   79.84%           
=======================================
  Files          35       35           
  Lines        3457     3458    +1     
  Branches      576      576           
=======================================
+ Hits         2760     2761    +1     
  Misses        560      560           
  Partials      137      137

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

azure-pipelines · 2024-06-11T10:47:18Z

Azure Pipelines successfully started running 1 pipeline(s).

sk1p · 2024-06-11T13:01:39Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-11T13:01:50Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2024-06-21T13:16:06Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2024-06-21T13:19:57Z

Azure Pipelines successfully started running 1 pipeline(s).

matbryan52 · 2024-06-21T13:35:46Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-21T13:35:55Z

Azure Pipelines successfully started running 1 pipeline(s).

matbryan52 · 2024-06-21T14:30:57Z

I am genuinely perplexed about this, without installing uv or tox-uv, and forcing also tox env recreation the pipeline is still using uv to install packages, and I think this is referencing a py38 install which we never ask for.

If there is some kind of caching on the pipelines maybe good to reset everything? I don't think I have the access to do that myself.

The numba test troubles are equally odd, I tried on ptycho with the same python, environment (checked via diff), numpy 2, branch, and there is no trouble. It looks like it only affects 16 bits per pixel tests in merlin/.../test_encode_roundtrip..., independently of the encode/decode size. There aren't even any warnings emitted ! All I can guess is that it it due to numba caching somewhere?

sk1p · 2024-06-25T13:55:36Z

I am genuinely perplexed about this, without installing uv or tox-uv, and forcing also tox env recreation the pipeline is still using uv to install packages,

This might be because the venv is not forcibly removed if it already exists, and tox-uv would still possibly be installed in there.

and I think this is referencing a py38 install which we never ask for.

Yeah, that's the really weird one. The Python 3.8 is just one of the versions that is made available to the azure agents. I'm not sure yet, and I have no idea why uv picks that one up! Maybe because that's the "native" version of the ubuntu:20.04 image we use as a base?

We might want to move to 22.04 as a base image for the azure agent, which does still support the Python versions we need, as far as I can see.

If there is some kind of caching on the pipelines maybe good to reset everything? I don't think I have the access to do that myself.
The numba test troubles are equally odd, I tried on ptycho with the same python, environment (checked via diff), numpy 2, branch, and there is no trouble. It looks like it only affects 16 bits per pixel tests in merlin/.../test_encode_roundtrip..., independently of the encode/decode size. There aren't even any warnings emitted ! All I can guess is that it it due to numba caching somewhere?

I didn't look further yet, but an issue is that the azure agent doesn't completely clean up the environment after a run, so there may indeed be some files that stick around (like tox environments, the venv, and the cache of pip/uv). I just re-created the containers, and I'll re-run it again, to see if that changed anything.

sk1p · 2024-06-25T13:55:44Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-25T13:55:55Z

Azure Pipelines successfully started running 1 pipeline(s).

sk1p · 2024-06-25T14:03:27Z

The numba test troubles are equally odd, I tried on ptycho with the same python, environment (checked via diff), numpy 2, branch, and there is no trouble. It looks like it only affects 16 bits per pixel tests in merlin/.../test_encode_roundtrip..., independently of the encode/decode size. There aren't even any warnings emitted ! All I can guess is that it it due to numba caching somewhere?

Hum... I can reproduce the failures - they only happen in the non-jit case (NUMBA_DISABLE_JIT=1 - via tox -e numba_coverage for example). I had to do some changes in LiberTEM itself for the offline decoding functions - probably there's an issue with changed dtype promotion rules (numba doesn't fully implement these yet, which is why they only fail with the jit disabled)

Hopefully that will be cleaned up by the agent

sk1p · 2024-06-25T14:09:24Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-25T14:09:34Z

Azure Pipelines successfully started running 1 pipeline(s).

matbryan52 · 2024-06-25T14:18:03Z

Hum... I can reproduce the failures - they only happen in the non-jit case (NUMBA_DISABLE_JIT=1 - via tox -e numba_coverage for example). I had to do some changes in LiberTEM itself for the offline decoding functions - probably there's an issue with changed dtype promotion rules (numba doesn't fully implement these yet, which is why they only fail with the jit disabled)

Ah I have just been running with pytest ... -m with_numba as I have found tox difficult when I try to run it myself. So the jit'd functions pass the tests but not the python-only form 🤔

azure-pipelines-data.yml

sk1p · 2024-06-25T14:50:34Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-25T14:50:44Z

Azure Pipelines successfully started running 1 pipeline(s).

sk1p · 2024-06-25T14:54:14Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-25T14:54:24Z

Azure Pipelines successfully started running 1 pipeline(s).

sk1p · 2024-06-25T15:00:07Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-25T15:00:21Z

Azure Pipelines successfully started running 1 pipeline(s).

Numba and numpy 2.x don't yet agree on the dtype conversion rules, so for disabled numba, we need to explicitly upcast to np.uint16 at some places.

sk1p · 2024-06-25T15:16:24Z

/azp run libertem.libertem-live-data

azure-pipelines · 2024-06-25T15:16:33Z

Azure Pipelines successfully started running 1 pipeline(s).

matbryan52 · 2024-06-25T15:25:55Z

So weird that uv is fixated wrong python. Otherwise well done, LGTM!

sk1p · 2024-06-25T15:34:35Z

So weird that uv is fixated wrong python. Otherwise well done, LGTM!

Yeah, extra weird that it works fine for the other jobs, like data_tests or check_example_notebooks 🤷

Run docs checks in venv, like other jobs

f4752c6

sk1p added the infra CI/CD infrastructure etc. label Jun 11, 2024

sk1p changed the title ~~Run docs checks in venv, like other jobs~~ WIP: fix docs-check job Jun 11, 2024

Use uv for libertem_master_integration job

8b5d56f

This comment was marked as outdated.

Sign in to view

Explicitly activate the venv

27688c5

This comment was marked as outdated.

Sign in to view

matbryan52 added 2 commits June 21, 2024 14:23

Try forcing basepython for docs check

f9ed865

Revert, force localdir

3885f21

Without uv

d142bd9

With recreate

6b6fda0

Create venv in $(Agent.TempDirectory)

7e11e76

Hopefully that will be cleaned up by the agent

matbryan52 reviewed Jun 25, 2024

View reviewed changes

azure-pipelines-data.yml Outdated Show resolved Hide resolved

Switch recreate_tox to old default; try uv again

c7babad

Give up on uv for now

692a369

Make sure the old venv doesn't exist anymore in the CI env

49f3ab7

Fixes for the merlin decoders

acae5ff

Numba and numpy 2.x don't yet agree on the dtype conversion rules, so for disabled numba, we need to explicitly upcast to np.uint16 at some places.

sk1p changed the title ~~WIP: fix docs-check job~~ Fix docs-check job; numpy 2.x fixes Jun 25, 2024

matbryan52 approved these changes Jun 25, 2024

View reviewed changes

matbryan52 merged commit a97f501 into LiberTEM:master Jun 25, 2024
31 checks passed

sk1p deleted the docs-check-in-venv branch June 25, 2024 15:32

sk1p added this to the 0.3 milestone Oct 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `docs-check` job; numpy 2.x fixes #157

Fix `docs-check` job; numpy 2.x fixes #157

sk1p commented Jun 11, 2024 •

edited

Loading

sk1p commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

codecov bot commented Jun 11, 2024 •

edited

Loading

This comment was marked as outdated.

azure-pipelines bot commented Jun 11, 2024

This comment was marked as outdated.

sk1p commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

azure-pipelines bot commented Jun 21, 2024

azure-pipelines bot commented Jun 21, 2024

matbryan52 commented Jun 21, 2024

azure-pipelines bot commented Jun 21, 2024

matbryan52 commented Jun 21, 2024 •

edited

Loading

sk1p commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

matbryan52 commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

matbryan52 commented Jun 25, 2024 •

edited

Loading

sk1p commented Jun 25, 2024

Fix docs-check job; numpy 2.x fixes #157

Fix docs-check job; numpy 2.x fixes #157

Conversation

sk1p commented Jun 11, 2024 • edited Loading

Contributor Checklist:

Reviewer Checklist:

sk1p commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

codecov bot commented Jun 11, 2024 • edited Loading

Codecov Report

This comment was marked as outdated.

azure-pipelines bot commented Jun 11, 2024

This comment was marked as outdated.

sk1p commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

azure-pipelines bot commented Jun 21, 2024

azure-pipelines bot commented Jun 21, 2024

matbryan52 commented Jun 21, 2024

azure-pipelines bot commented Jun 21, 2024

matbryan52 commented Jun 21, 2024 • edited Loading

sk1p commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

matbryan52 commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

sk1p commented Jun 25, 2024

azure-pipelines bot commented Jun 25, 2024

matbryan52 commented Jun 25, 2024 • edited Loading

sk1p commented Jun 25, 2024

Fix `docs-check` job; numpy 2.x fixes #157

Fix `docs-check` job; numpy 2.x fixes #157

sk1p commented Jun 11, 2024 •

edited

Loading

codecov bot commented Jun 11, 2024 •

edited

Loading

matbryan52 commented Jun 21, 2024 •

edited

Loading

matbryan52 commented Jun 25, 2024 •

edited

Loading