Linking Cohorts and Datasets

[redmitry]

Hi all,

Although we have the nice general model schema in the documentation: models, there is no standard link between models and Cohorts & Datasets defined.

The CINECA synthetic dataset have non-formal "ids" property in the dataset.

Looking at the previous chart, IMO we must add

"cohortIds" : {
    "type":  "array",
    "items": {
        "type": "string"
     }
}

into the individual entity (the m..n individual -> cohort link) and then add

"datasetId" : {
    "type":  "string"
}

in the cohort (linking each cohort to the corresponding dataset).

Having this data, I would be able to check the GA4GH Visas either via cohort or dataset permissions.

Best,

Dmitry

[redmitry]

UPDATE:
Even though we have nice chart: beacon v2 model actual model has no the links (aka foreign keys). Also to ensure coherence these must be mutually exclusive.
I explain:
genomicVariations

{
  "properties": {
    "datasetId": { "type": "string"},
    "biosampleId": { "type": "string"},
    "individualId": { "type": "string"},
    "analysisId": { "type": "string"},
    "runId": { "type": "string"}
  },
  "oneOf": [
    { "not": {
        "anyOf": [
            { "required": ["biosampleId"] },
            { "required": ["individualId"] },
            { "required": ["analysisId"] },
            { "required": ["runId"] }
        ]
      }
    },
    { "required": ["biosampleId"] },
    { "required": ["individualId"] },
    { "required": ["analysisId"] },
    { "required": ["runId"] }
  ]
}

So here we may have at most one of "biosampleId", "individualId", "analysisId" or "runId" defined.
This would prevent us from referring, for example, biosampleId with individualId = X and have individualId = Y at the same time.

The same pattern should be applied to all entryTypes so we always have only one root to the Individual

Cheers,

D.

[mbaudis]

@redmitry I don't really follow. genomicVariations have CaseLevelVariant where all the ids are referenced. And yes, you cannot have 2 individuals (ids) for the same biosample (which makes sense; other cases with pooled samples etc. are not considered part of the standard model; pooled samples etc. could be considered their own "synthetic individual".

So, what is the problem? individual - 0..n -> biosample - 0..n -> analysis -> 0..n -> run -> 0..n -> variant (with shortcuts possible; e.g. no run).

And (at least as concept) the data is organised in datasets, and cohorts can span datasets (where I think the model lacks this specification and opinions might differ).

[redmitry]

We do not have links neither to "analysis" nor to "run", "biosample" or "individual" in the genomicVariations .

I propose:

1. add "datasetId" as we can always directly link variant to a dataset,
2. add a "choice" (mutually exclusive) links to "analysis", "run", "biosample" and "individual".

This way if we only have variants:
variant -> dataset
if we have individuals:
variant -> individual -> cohort(s) -> dataset(s)

For the Paasports/Visas I need to filter variants by datasets and currently have no this in a schema.

Dmitry

[mbaudis]

I very much agree that there is no place where cohort memberships are defined. There is just an endpoint to retrieve the individuals of a given cohort (/cohorts/{id}/individuals) - again only in OpenAPI?! - but no place in the schemas to provide that link.

(Also, IMO, the concept of defining cohorts as collections of individuals is flawed since in more complex scenarios you might have a database of individuals but they might be part of different studies etc., so not all their data might be part of every collection).)

Looking at the previous chart, IMO we must add cohortIds

IMO the wrong way around; cohorts should have individualIds.

[mbaudis]

@redmitry

We do not have links neither to "analysis" nor to "run", "biosample" or "individual" in the [genomicVariations ]

No, we have. CaseLevelVariant

[redmitry]

$${\color{red}BOLD}$$ $${\color{red}ISSUE}$$ 🤦‍♂️

That is a ruin for performance.

I found an issue in EGA:
EGA-archive/beacon2-ri-tools#15

While for a clinician the concept "a cohort comprises a set of individuals", data modelling means individuals have cohortId.
Imagine B1MG - we just couldn't model it this way.
We have a document which is undividable. Mongo has a 16Mb limit but even we do not exceed it - adding new individual with variant X means entirely replace document.

Best,

Dmitry

[mbaudis]

@redmitry This is data model vs. implementation. But as @mrueda mentions in the referenced issue it's up to implementers how to solve this (external lookup etc.). So as a concept a cohort should have a list of the individuals ... etc.; and if it doesn't fit into a db field you can imagine that every cohort could have its own dedicated "table" or that you have a a central table for each cohort x individual entry or that you save all cohortId values with each sample or individual (actually our approach) but "off schema" etc.

I'm actually not too much of a purist so any practical solution works for me. But logically the cohort should define the individuals or samples.

[redmitry]

I must be from a USSR school, but we do not need to have individuals in the cohort object to get them.
For this we have the /cohorts/{id}/individuals

D.

[mbaudis]

Что ты знаешь...¹ As mentioned above, this is just an endpoint but is not reflected in the model (neither cohortIds in individual model nor individualIds in cohort model).

Footnotes

1. Just a victim of GDR education system... ↩

[costero-e]

Generally, I like the solution proposed by @redmitry, adding a datasetId and cohortIds, and I agree that this would solve the problem enough. On the other hand, I wouldn't make at least one id required:

  "properties": {
    "datasetId": { "type": "string"},
    "biosampleId": { "type": "string"},
    "individualId": { "type": "string"},
    "analysisId": { "type": "string"},
    "runId": { "type": "string"}
  },
  "oneOf": [
    { "not": {
        "anyOf": [
            { "required": ["biosampleId"] },
            { "required": ["individualId"] },
            { "required": ["analysisId"] },
            { "required": ["runId"] }
        ]
      }
    },
    { "required": ["biosampleId"] },
    { "required": ["individualId"] },
    { "required": ["analysisId"] },
    { "required": ["runId"] }
  ]
}

Maybe there is a case in which you don't want to relate the data you have to a specific dataset or cohort or you don't have this information. Plus, I don't see why we want to restrict a variant relationship to only one id. I know that programatically this will lead to less mistakes, but on the other hand, we are letting users have less information on which is the id of every entry for the variant found (which can be useful for data research). I know this can lead to mistakes but it's more a problem of data curation than for beacon developers.
I would just add the datasetId and cohortIds as optional fields in properties.

[mbaudis]

I'm still not sure what you're going at here (I do understand to provide datasetId or datasetIds query parameters, same for cohorts).

But both these can be managed externally; i.e. having a lookup collection for individuals, cohorts ... etc.

And for variants, store them as individual calls where then every call has its associated analysis_id and everything upstream can be derived.

In our example we have in MongoDB datasetId == databaseName; and have samples or individuals tagged for the cohorts they belong to (a better way probbaly would be a separate lookup).

So +1 for a datasetIds as query parameter etc., but I wouldn't model them into entities. IMO