Add CI to build mulled containers with Wave #4080

edmundmiller · 2023-10-17T12:31:39Z

maxulysse · 2023-10-18T07:55:25Z

Why don't we do that already?

edmundmiller · 2023-10-18T08:24:14Z

Why don't we do that already?

Just hadn't gotten around to it 😆

I'm also having trouble authenticating with the registry. I didn't think it would have to go through tower if it was public. @pditommaso could you confirm?

I think if not we can use Tower to build the images in CI using nextflow inspect if the Dockerfile or environment.yml changes and push the image for people.

tests/config/nextflow.config

MatthiasZepper · 2023-10-18T09:05:54Z

How does that impact offline use of pipelines? A mulled container can easily be downloaded with nf-core download. How about those built with Wave? Will this still work?

ewels · 2023-10-18T09:08:40Z

modules/nf-core/bowtie/align/environment.yml

+  - bioconda::bowtie=1.3.0
+  - bioconda::samtools=1.16.1


bioconda:: probably not needed.

This is a tricky point, to include package compatible with ARM64, we are maintaining a seqera channel forked from bioconda. Therefore it be useful to add seqera in the channels and remove the bioconda:: prefix.

However, once bioconda will support ARM64, the goal is to merge those recipes there.

edmundmiller · 2023-10-18T10:08:48Z

How does that impact offline use of pipelines? A mulled container can easily be downloaded with nf-core download. How about those built with Wave? Will this still work?

That was @ewels concern as well! I think nextflow inspect might be able to handle this in theory.

But I think I misunderstood that the wave builds are happening locally, but I believe they're happening on a web server.

Ideally this would just be for building the containers, and then end users will be able to pull the container normally, unless they modify the environment.

adamrtalbot · 2023-10-18T10:56:47Z

I believe this is a supported feature of Wave Freeze and nextflow inspect -concretize, however I think if it can't run offline this feature is a no-go, so we should triple check.

Build time is happening at the Wave server, but this should be functionally equivalent to running nf-core download. @MatthiasZepper, would you agree?

MatthiasZepper · 2023-10-18T12:50:06Z

I am not in Barcelona and probably somewhat out of the loop. Additionally, I have no experience using Wave, @mribeirodantas Bytesize talk is essentially all I know about it.

In general, I have no objections against abandoning Bioconda's CI in favour of Wave. Coincidentally, I learned yesterday why Oxford Nanopore has little confidence in Bioconda and that they are strongly favouring own distribution channels. Moving away from Bioconda allows ONT to support AArch64 natively and for a better provenance in dependencies. Taking into account the importance of ONT software in nf-core pipelines, I think that alone justifies allowing for other software sources and container registries in modules, including building them with Wave.

I believe this is a supported feature of Wave Freeze and nextflow inspect -concretize, however I think if it can't run offline this feature is a no-go, so we should triple check.

The ability to run offline is indeed paramount for us at NGI. We would need to maintain our own forks of the pipelines with patched modules, if that option was dropped from the nf-core pipelines/modules. From which registry the container is downloaded/pulled is pretty much irrelevant for us (a non-free/non-sponsored service might be a slight downer, though), but might matter for scientists in other countries that are under international sanctions (e.g. Iran, Russia etc.?)

I believe this is a supported feature of Wave Freeze and nextflow inspect -concretize. Build time is happening at the Wave server, but this should be functionally equivalent to running nf-core download.

Tools 2.10 is simply extracting the container definitions from the modules & configs with Regexes. Subsequently, it is downloading all that start with http:// paths and pulling the remainder. I am not aware of any code that would support Wave builds as of now.

But the decision of abandoning the regexes in favour of nextflow inspect for future releases is set in stone and I presume you are actually implementing that already @adamrtalbot & Marcel? In that case, nf-core download could probably also be enabled to resolve any Wave instructions and also kick off builds and obtain images from a dynamically created container URI if needed?

My only concern is that the pipeline, when executed, will not find the appropriate container without contacting the Wave server asking for its hash respectively the required layers. But I believe this concern is actionable by the Nextflow/Wave plugin developers?

pditommaso · 2023-10-18T16:08:08Z

tests/config/nextflow.config

+    wave.enabled = true
+    wave.strategy = ['dockerfile', 'conda', 'container']


Suggested change

wave.enabled = true

wave.strategy = ['dockerfile', 'conda', 'container']

wave.enabled = true

wave.strategy = ['conda', 'container']

wave.freeze = true

wave.build.repository = 'docker.io/emiller88/modules'

Singularity native requires the freeze mode, which in turn requires also the target build repository

pditommaso · 2023-10-18T16:09:16Z

tests/config/nextflow.config

@@ -28,6 +30,9 @@ if ("$PROFILE" == "singularity") {
    docker.userEmulation = false
    docker.fixOwnership = true
    docker.runOptions = '--platform=linux/amd64 -u $(id -u):$(id -g)'
+    wave.enabled = true
+    wave.strategy = ['dockerfile', 'conda', 'container']


Suggested change

wave.strategy = ['dockerfile', 'conda', 'container']

wave.strategy = 'conda'

Likely you want only rely on conda packages

We have a few modules with dockerfiles (bclconvert off the top of my head) and a few that use images only and don't have conda packages (parabricks I think is this way)

pditommaso · 2023-10-18T16:10:41Z

tests/config/nextflow.config

@@ -28,6 +30,9 @@ if ("$PROFILE" == "singularity") {
    docker.userEmulation = false
    docker.fixOwnership = true
    docker.runOptions = '--platform=linux/amd64 -u $(id -u):$(id -g)'
+    wave.enabled = true
+    wave.strategy = ['dockerfile', 'conda', 'container']
+    wave.build.repository = 'docker.io/emiller88/modules'


This is not strictly necessary. if omitted it used Wave default registry (tho images will only be cached for one week)

Ideally we'd want to not rebuild them unless the environment changes.

The other issue is it would make more sense to have one container repo per module.

Currently it looks like wave creates a new tag with the env name rather than a image per env.

modules/nf-core/bowtie/align/main.nf

pditommaso · 2023-10-18T16:22:41Z

I'm also having trouble authenticating with the registry. I didn't think it would have to go through tower if it was public. @pditommaso could you confirm?

Currently, without credentials you will be limited to 25 builds per day. We are discussing how to allow a community registry for nf-core pipelines.

MatthiasZepper · 2023-10-19T09:32:22Z

Currently, without credentials you will be limited to 25 builds per day. We are discussing how to allow a community registry for nf-core pipelines.

Would a combination of Wave with Kraken be an option? There are enough academic institutions represented in the nf-core community that could sponsor a server/mirror or two, so we could host the images in a peer-to-peer network after they were build with Wave? Unfortunately, this does seemingly not support Singularity images...

This reverts commit 69e1ea5.

This reverts commit 5a3d546.

ewels · 2024-06-18T20:27:15Z

modules/nf-core/bowtie/align/main.nf

    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
-        'https://depot.galaxyproject.org/singularity/mulled-v2-ffbf83a6b0ab6ec567a336cf349b80637135bca3:c84c7c55c45af231883d9ff4fe706ac44c479c36-0' :
-        'biocontainers/mulled-v2-ffbf83a6b0ab6ec567a336cf349b80637135bca3:c84c7c55c45af231883d9ff4fe706ac44c479c36-0' }"
+        'oras://community.wave.seqera.io/library/bowtie_samtools:16f00b34cfc72399' :


I think this will break nf-core download. I'd prefer the https links if possible, and should test the download if we can.

Is there a Seqera feedback link for the https links?

modules/nf-core/bowtie/align/environment.yml

ewels · 2024-06-18T20:27:54Z

modules/nf-core/bowtie/align/main.nf

@@ -3,9 +3,10 @@ process BOWTIE_ALIGN {
    label 'process_high'

    conda "${moduleDir}/environment.yml"
+    container ""


Accidental?

askimed/nf-test#258

ewels

We should add in ARM builds as well (may need to tolerate + log failures).

Also, is it possible to parallelise the different requests in separate jobs? Would go 4x faster then.

ewels · 2024-09-18T14:22:35Z

.github/workflows/wave.yml

+      - name: Create a registry name
+        uses: actions/github-script@v7
+        id: registry-name
+        with:
+          result-encoding: string
+          script: |
+            return '${{ matrix.files }}'.replace('modules/nf-core/', '').replace('/environment.yml', '').replace('/', '_');


This is to have name in the conda environment, right? We can remove if so.

No, this was to have a different wave repo per module. We can keep this for anything built by a Dockerfile for now(bclconvert, cellranger, etc.)

ewels · 2024-09-18T14:25:46Z

It'd be nice to also write the container URIs to the meta.yml file. No problem for it to be duplicated for now, means we can just remove from the main.nf file later. See also #6659 (comment)

Note that we haven't decided on a spec for how this should look in the meta YAML file yet, so need to do that. Should be easy to agree though, I have a rough idea in my head already.

edmundmiller · 2024-09-20T23:28:12Z

We should add in ARM builds as well (may need to tolerate + log failures).

Sounds good! We can log it in the CI definitely, or maybe we output it to files like bioconda does.

Also, is it possible to parallelise the different requests in separate jobs? Would go 4x faster then.

Already is! It just skips the step depending on the matrix.

edmundmiller · 2024-11-07T15:59:28Z

Going to merge this finally. There isn't anything in here that should break anything.

Things that ended up in this PR:

Requests images on environment.yml changes
Switched bowtie modules and samtools/view to Seqera containers(which we've been doing)
Added Renovate comments

This is so we can start making some smaller PRs to the system and testing automations out.

mahesh-panchal · 2024-11-08T07:30:49Z

I'm not sure if it's related to this PR in particular, but I'm getting notifications of wave workflow run failed.

https://github.com/nf-core/modules/actions/runs/11737926211

adamrtalbot · 2024-11-08T12:29:37Z

I'm not sure if it's related to this PR in particular, but I'm getting notifications of wave workflow run failed.

https://github.com/nf-core/modules/actions/runs/11737926211

Looks like missing values.

Secrets only work if the PR originates from the same organisation (in this case it would need to come from nf-core/modules).

edmundmiller · 2024-11-08T17:00:17Z

Yeah, thanks @mahesh-panchal fixed that in #6954 (hopefully)

edmundmiller self-assigned this Oct 17, 2023

edmundmiller force-pushed the wave branch from 17a2d31 to a3819cc Compare October 17, 2023 12:45

edmundmiller requested review from a team and mirpedrol October 17, 2023 13:43

maxulysse approved these changes Oct 17, 2023

View reviewed changes

edmundmiller force-pushed the wave branch from a3819cc to 76b2b3a Compare October 17, 2023 14:30

edmundmiller requested review from pditommaso, maxulysse and bentsherman October 17, 2023 14:32

edmundmiller changed the title ~~Build containers with wave~~ Replace mulled containers with wave Oct 17, 2023

edmundmiller requested a review from adamrtalbot October 17, 2023 16:23

edmundmiller marked this pull request as ready for review October 18, 2023 08:24

edmundmiller requested a review from a team as a code owner October 18, 2023 08:24

edmundmiller requested a review from drpatelh October 18, 2023 08:24

adamrtalbot reviewed Oct 18, 2023

View reviewed changes

tests/config/nextflow.config Outdated Show resolved Hide resolved

ewels reviewed Oct 18, 2023

View reviewed changes

pditommaso reviewed Oct 18, 2023

View reviewed changes

modules/nf-core/bowtie/align/main.nf Outdated Show resolved Hide resolved

edmundmiller requested a review from KevinMenden as a code owner October 19, 2023 16:26

edmundmiller force-pushed the wave branch from fec27b8 to 385dcd4 Compare November 2, 2023 20:14

edmundmiller force-pushed the wave branch from aa96c61 to a1e54b4 Compare June 18, 2024 19:14

edmundmiller added 2 commits June 18, 2024 14:17

Revert "build: Bump Nextflow version requirement"

8fa7b0d

This reverts commit 69e1ea5.

Revert "test: Add ociAutoPull to nf-test"

727dda0

This reverts commit 5a3d546.

edmundmiller force-pushed the wave branch from 79a88b5 to 727dda0 Compare June 18, 2024 19:19

edmundmiller mentioned this pull request Jun 18, 2024

Conda lock files #5827

Open

ewels reviewed Jun 18, 2024

View reviewed changes

edmundmiller added 3 commits September 10, 2024 12:01

test(#6505): Snapshot the versions contents, not the hash

daa44fe

ci(#6505): Update version snapshot after building containers

457bf9e

test(#6505): Attempt a topic channel with tests

2925c17

askimed/nf-test#258

ewels reviewed Sep 18, 2024

View reviewed changes

MatthiasZepper mentioned this pull request Sep 24, 2024

Download: Fix support for Seqera Community Containers nf-core/tools#3179

Open

edmundmiller added 4 commits October 30, 2024 11:52

Merge remote-tracking branch 'origin/master' into wave

455bc2d

chore: Bump to 1.5.0

2462a11

fix: Remove shard and filter on test bumping

6934f0b

build: Bump images to match environment

3fa2c06

edmundmiller enabled auto-merge October 30, 2024 11:53

ci: Fix nf-test setup

8c73e71

edmundmiller force-pushed the wave branch from 091cdcb to 8c73e71 Compare October 30, 2024 14:13

ci: Remove snapshot bumping

24de614

edmundmiller force-pushed the wave branch from 959420a to 24de614 Compare October 31, 2024 17:51

build: Fix containers in bowtie

86ad692

edmundmiller disabled auto-merge November 7, 2024 15:59

edmundmiller merged commit 669eb24 into master Nov 7, 2024
13 checks passed

edmundmiller deleted the wave branch November 7, 2024 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI to build mulled containers with Wave #4080

Add CI to build mulled containers with Wave #4080

edmundmiller commented Oct 17, 2023 •

edited

Loading

maxulysse commented Oct 18, 2023

edmundmiller commented Oct 18, 2023

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

ewels Oct 18, 2023

pditommaso Oct 18, 2023

edmundmiller commented Oct 18, 2023

adamrtalbot commented Oct 18, 2023 •

edited

Loading

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

pditommaso Oct 18, 2023

pditommaso Oct 18, 2023

edmundmiller Oct 18, 2023

pditommaso Oct 18, 2023

edmundmiller Oct 18, 2023

pditommaso commented Oct 18, 2023

MatthiasZepper commented Oct 19, 2023 •

edited

Loading

ewels Jun 18, 2024

edmundmiller Sep 25, 2024

ewels Jun 18, 2024

ewels left a comment

ewels Sep 18, 2024

edmundmiller Sep 20, 2024

ewels commented Sep 18, 2024

edmundmiller commented Sep 20, 2024 •

edited

Loading

edmundmiller commented Nov 7, 2024

mahesh-panchal commented Nov 8, 2024

adamrtalbot commented Nov 8, 2024

edmundmiller commented Nov 8, 2024

		wave.enabled = true
		wave.strategy = ['dockerfile', 'conda', 'container']

	wave.strategy = ['dockerfile', 'conda', 'container']
	wave.strategy = 'conda'

Add CI to build mulled containers with Wave #4080

Add CI to build mulled containers with Wave #4080

Conversation

edmundmiller commented Oct 17, 2023 • edited Loading

maxulysse commented Oct 18, 2023

edmundmiller commented Oct 18, 2023

MatthiasZepper commented Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmundmiller commented Oct 18, 2023

adamrtalbot commented Oct 18, 2023 • edited Loading

MatthiasZepper commented Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pditommaso commented Oct 18, 2023

MatthiasZepper commented Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewels left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewels commented Sep 18, 2024

edmundmiller commented Sep 20, 2024 • edited Loading

edmundmiller commented Nov 7, 2024

mahesh-panchal commented Nov 8, 2024

adamrtalbot commented Nov 8, 2024

edmundmiller commented Nov 8, 2024

edmundmiller commented Oct 17, 2023 •

edited

Loading

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

adamrtalbot commented Oct 18, 2023 •

edited

Loading

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

MatthiasZepper commented Oct 19, 2023 •

edited

Loading

edmundmiller commented Sep 20, 2024 •

edited

Loading