Error "Can't stage file https://files.plutof.ut.ee:" Download fail

[himamura2]

Description of the bug

I am getting "Can't stage file" error for ITS dada_ref_taxonomy: 'unite-fungi'.

V=2.10.0
ITS analysis

"  task: name=NFCORE_AMPLISEQ:AMPLISEQ:FORMAT_TAXONOMY; work-dir=null
  error [nextflow.exception.ProcessStageException]: Can't stage file https://files.plutof.ut.ee/public/orig/1E/25/1E25CA4CC30A31C2E2B8CB2C89824C83D080A7F5A62E6263A0E95B37C6628067.tgz -- reason: Unable to access path: https://files.plutof.ut.ee/public/orig/1E/25/1E25CA4CC30A31C2E2B8CB2C89824C83D080A7F5A62E6263A0E95B37C6628067.tgz
Jul-22 10:29:23.571 [Actor Thread 36] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:FORMAT_TAXONOMY'

Caused by:
  Can't stage file https://files.plutof.ut.ee/public/orig/1E/25/1E25CA4CC30A31C2E2B8CB2C89824C83D080A7F5A62E6263A0E95B37C6628067.tgz -- reason: Unable to access path: https://files.plutof.ut.ee/public/orig/1E/25/1E25CA4CC30A31C2E2B8CB2C89824C83D080A7F5A62E6263A0E95B37C6628067.tgz
"

I contacted "plutof.ut.ee team" about the error and they said:

I think you have to ask for new URL-s for these files. Last week we closed our legacy file system. I think your files were part of it.
In our new file repository solution, you can have more control withe the naming of the files.
Sorry for the trouble.
All the best,
Allan & PlutoF team

Is this file controled by ampliseq team or dada2 team?
best,
Hideo

Command used and terminal output

No response

Relevant files

file:///home/himamura/ITS_nextflow.log
0:-26:690:44

System information

Nextflow version (eg. 23.10.0)
https://github.com/nf-core/ampliseq/tree/2.10.0

[d4straub]

Thanks for the report and attempting to find a solution!
I adjusted the formatting of your post that its a little easier to read.

It is very unfortunate that they changed their file system, this will of course break the links that are implemented into the pipeline.
For now, if you find any other source of that files, you could use --dada_ref_tax_custom and optionally also --dada_ref_tax_custom_sp.

Is this file controled by ampliseq team or dada2 team?

As far as I know its not controlled by "us".

Because I am not overly familiar with that file system, could you let me know if you find the files under other links?

[himamura2]

Thank you for the really quick reponse. I thought this was the microsoft issue last week but it did not resolved. About the file, I downloaded unit-fungi=9 before. And I copied the file to the location where it should be. But I am getting the error that the file cannot be downloaded.
ITS_nextflow.log
So, I do not know how to get around this issue. Likely I should post this to DADA2 team, since they may be able to update the file and resolve the issue.
I saw your suggestion now. I will try that option.
best,
Hideo

[d4straub]

@himamura2 thanks for posting in the DADA2 repo, but I think its an issue of plutof.ut.ee, not DADA2.

Hi @erikrikarddaniel @jtangrot any idea?

[erikrikarddaniel]

@himamura2 thanks for posting in the DADA2 repo, but I think its an issue of plutof.ut.ee, not DADA2.

Hi @erikrikarddaniel @jtangrot any idea?

I think our urls, i.e. in the Ampliseq ref_databases.config might need to be updated if the PlutoF people moved them. This is probably easiest for @jtangrot , but I'll have a look.

[gagnedavid]

Hi,
I have almost the same error:
WARN: Unable to stage foreign file:
https://files.plutof.ut.ee/public/orig/7B/23/7B235835FAF5C85D7B01E40FEF17F687914CB81A182554C5BD95E3168328E604.tgz

[erikrikarddaniel]

I just merged PR #766 that tries to fix this issue into dev. You can try it by running Nextflow with -r dev. I'm closing in the hope that it works, but please reopen (with a comment) if the problem persists for you.

[himamura2]

Thank you Daniel. -r dev fixed the issuee for 16S and ITS.