Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix endpoint for TRS #15

Closed
kinow opened this issue Jul 1, 2022 · 6 comments
Closed

Fix endpoint for TRS #15

kinow opened this issue Jul 1, 2022 · 6 comments

Comments

@kinow
Copy link

kinow commented Jul 1, 2022

I am experimenting running WfExS (cc @jmfernandez) with Yevis/DDBJ. Here's what I did so far:

# file: tests/yevis-test.yaml
trs_endpoint: https://ddbj.github.io/workflow-registry/
workflow_id: 0d2ae4c2-fe4c-48f7-811a-ac277776533e
version: 1.0.0
workflow_config:
  secure: false
# All the inputs must be URLs or CURIEs from identifiers.org
params:
outputs:
  • Tried running the workflow python WfExS-backend.py execute -W tests/yevis-test.yaml

It had failed with an error about toolclass vs. tool_class, already fixed by @suecharo (thanks!).

But now it is failing with another error:

(venv) kinow@ranma:~/Development/python/workspace/WfExS-backend$ python WfExS-backend.py execute -W tests/yevis-test.yaml
2022-07-01 14:08:43,132 - [ERROR] FUSE filesystem command encfs not found. Please install it if you are going to use a secured staged workdir
* Command "execute".
	- Working directory will be /tmp/WfExS-workdirlu5yc1pnbackend/fade65fc-bee8-4ea0-8aa3-9b595a6c0c71
	- Instance fade65fc-bee8-4ea0-8aa3-9b595a6c0c71 (nickname 'handled cider') (to be used with -J)
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/WfExS-backend/WfExS-backend.py", line 568, in <module>
    main()
  File "/home/kinow/Development/python/workspace/WfExS-backend/WfExS-backend.py", line 550, in main
    stagedSetup = wfInstance.stageWorkDir()
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/workflow.py", line 1254, in stageWorkDir
    self.fetchWorkflow()
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/workflow.py", line 715, in fetchWorkflow
    i_workflow = self.getWorkflowRepoFromTRS(offline=offline)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/workflow.py", line 2271, in getWorkflowRepoFromTRS
    _, trsFilesDir, trsFilesMeta, _ = self.wfexs.cacheFetch(INTERNAL_TRS_SCHEME_PREFIX + ':' + toolFilesURL, CacheType.TRS, offline)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/wfexs_backend.py", line 852, in cacheFetch
    return self.cacheHandler.fetch(
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/cache_handler.py", line 638, in fetch
    raise nested_exception
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/cache_handler.py", line 619, in fetch
    raise che from nested_exception
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/cache_handler.py", line 582, in fetch
    inputKind, fetched_metadata_array, fetched_licences = schemeHandler(the_remote_file, tempCachedFilename, secContext=usableSecContext if usableSecContext else None)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/fetchers/trs_files.py", line 102, in fetchTRSFiles
    _ , metaprimary, _ = fetchClassicURL(file_url, descriptorMeta)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/fetchers/__init__.py", line 213, in fetchClassicURL
    raise FetcherException("Error fetching {} : {} {}".format(orig_remote_file, he.code, he.reason))
wfexs_backend.fetchers.FetcherException: Error fetching https://ddbj.github.io/workflow-registry/tools/0d2ae4c2-fe4c-48f7-811a-ac277776533e/versions/1.0.0/CWL/descriptor/https://zenodo.org/api/files/2422dda0-1bd9-4109-aa44-53d55fd934de/download-sra.cwl : 404 Not Found

Thanks! 👍

@suecharo
Copy link
Collaborator

suecharo commented Jun 2, 2023

Hello @kinow -san, I apologize for the delayed response. This error appears to occur because WfExS-backend.py is expecting to be able to fetch the file itself at /tools/{id}/versions/{version_id}/{type}/descriptor/{relative_path} in TRS.

As stated at https://editor.swagger.io/?url=https://raw.githubusercontent.com/ga4gh/tool-registry-schemas/develop/openapi/openapi.yaml, "Descriptors can often include imports that refer to additional descriptors. This returns additional descriptors for the specified tool in the same or other directories that can be reached as a relative path. This endpoint can be useful for workflow engine implementations like cwltool to programmatically download all the descriptors for a tool and run it. This can optionally include other files described with FileWrappers such as test parameters and containerfiles.", the original TRS definition indeed expects that behavior.

However, in yevis, due to the specification that returns TRS responses via GitHub pages, we are unable to implement this behavior. Instead, we're including the download link in the path of the /files response.

Given yevis's constraints of not wanting to have a REST API server (or any server resource), this is quite a predicament. Any suggestions on how we could navigate this issue would be much appreciated. (@inutano)

@kinow
Copy link
Author

kinow commented Jun 2, 2023

Thanks for the reply @suecharo , no worries about delay 🙆‍♂️

Given yevis's constraints of not wanting to have a REST API server (or any server resource), this is quite a predicament.

Hmm, can't think of any suggestion on how to workaround this issue. Maybe we can just document this for users of the endpoint?

Thanks!

@jmfernandez
Copy link

Hi @kinow (and @suecharo) ! If you provide me the WfExS-backend staging declaration file I can try figuring out a way to either detect or circumvent this kind of non-standard case in WfExS code.

@kinow
Copy link
Author

kinow commented Jun 2, 2023

Hi @jmfernandez ! 👋

The WfExS file I used was this one from the top comment:

# file: tests/yevis-test.yaml
trs_endpoint: https://ddbj.github.io/workflow-registry/
workflow_id: 0d2ae4c2-fe4c-48f7-811a-ac277776533e
version: 1.0.0
workflow_config:
  secure: false
# All the inputs must be URLs or CURIEs from identifiers.org
params:
outputs:

Not sure if the syntax changed. The last time I tried it it was months ago. I think it'd be useful if you can find a way to circumvent or give some feedback to users about the response not matching the expected standard.

jmfernandez added a commit to inab/WfExS-backend that referenced this issue Jun 3, 2023
DDBJ workflow registry TRS endpoint returns URLs instead of TRS files.
This commit tries circumventing the arosen issues in WfExS code.
@jmfernandez
Copy link

I have just pushed a fix to the main branch of WfExS-backend. Hope this helps!

@inutano
Copy link
Contributor

inutano commented Jun 5, 2023

Thank you @kinow @suecharo and @jmfernandez !! I understand this is the limitation of our strategy to implement a registry with gh pages, a fully static website. I'm closing this, but as @kinow suggested, at least we need to mention this in our documentation. Thanks for your testing and additional contribution @kinow @jmfernandez 🙏 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants