-
Notifications
You must be signed in to change notification settings - Fork 3
Home
This is a wiki space for the GA4GH Cloud Work Stream. Our group focuses on API standards (and implementations from our partner Driver projects and other community members) that make it easier to "send the algorithms to the data". Specifically, we have 4 API standards that allow you to share tools/workflows (TRS), execute individual jobs on clouds using a standard API (TES), run full CWL/WDL workflows on execution platforms (WES), and read/write data objects across clouds in an agnostic way (DRS).
These standards are really inspired by large-scale, distributed compute projects including, for example, PCAWG. Efforts such as these are characterized by data living in many different cloud environments, compute needing to be done across these cloud locations, and a motivation for working with disparate clouds via common and consistent API interfaces. The net effect is a highly portable analysis code that ultimately enables "FAIR" science, e.g. findable, accessible, interoperable, and reproducible tools, workflows, and datasets.
See this presentation from March of 2019 for an overview of what we do.
Our API standards are defined in OpenAPI 3.0 in repositories within the GA4GH GitHub organization:
- Data Repository Service API - DRS API
- Tool Registry Service API - TRS API
- Task Execution Service API - TES API
- Workflow Execution Service API - WES API
If a security issue with any of the above specifications is realised please send an email to security-notification@ga4gh.org detailing your concerns.
The way to contribute development effort and code to the project is via GitHub pull requests. GitHub provides a nice overview on how to create a pull request. See the CONTRIBUTING.md document in each schema repo. We follow HubFlow which means we use a feature branch strategy with pull requests always going to develop and releases happening from master.
See individual schema repos for more details. In short, the GA4GH has a number of Driver Projects, each of those associated with the Cloud Work Stream will nominate a representative. When a PR is up for a vote the Driver Project representatives will be notified over email and a note made in the PR. This starts a timer until the PR is accepted or rejected. None of the Driver representatives may vote against a proposed change for it to proceed. In addition, they must not be overridden by the Cloud Work Stream Leads, Brian O'Connor and David Glazer.
See this document that describes our plans for the next couple years.
We have a weekly call on Mondays at variable times to accommodate various timezones:
- 10am Pacific time on the second and third Monday of every month
- 2pm Pacific time on the last Monday of every month
We invite anyone interested in these standards and/or the systems that implement them to join us on these calls.
- Slack for real-time conversations, keep decisions and important info in GitHub issues please.
- Google Group for discussion and meeting announcements
- Current Agendas and notes
- 2018 Jan-Oct Agendas and notes, 2017 Agendas and notes, old Agendas and notes from the Containers and Workflows Task Team)
- Upcoming talks: please email Jeremy Adams if you want to get on the weekly agenda. Check the Cloud Meeting topics spreadsheet for slots
Dial-in details for the calls are available on request to Jeremy Adams
Our yearly meeting focused on collaboration across Work Streams and Driver Projects and for contributors to advance work on the GA4GH Strategic Roadmap. For more information see the event website.
See the event page for more information
See Cloud @ GA4GH Connect 2019 Hinxton
See this site for more info.
This meeting happened Oct 15-17th and the Cloud Work Stream had a breakout session for most of the 15th.
For our agenda see here
For the general conference, agenda see here
If you were not able to attend you can watch the Plenary recording here
We are working closely with the DREAM Challenges to test our API standards and workflow sharing process. Essentially, we are attempting to demonstrate API and process FAIR-compliance. This is a multi-phase effort with the first two challenges focusing on tool and workflow portability and reproducibility.
We also work closely with our Driver Projects through "Test Beds" that are focused efforts to demonstrate conformance of API implementations with the specifications.
Ultimately we want to have regular API verifications running and a registry where users can see the available services that are conforming to the specifications. This can be used by researchers to see what services are available and used by implementers to see what software is available for deployment that supports the GA4GH APIs.