Smart Proxy

Both server (a.k.a proxy) and client (a.k.a harverter) are automatically generated by swagger codegen based on OpenAPI descriptions.

Structure

There are several subprojects committed into this repository:

• codegen
  • openapi contains OpenAPI descriptors for proxy server/client and repositories (see below);
  • scripts contains scripts to generate source code based on OpenAPI descriptors from openapi folder;
• proxy-client contains source code for the harvester (all generated by codegen except controllers) that gathers data from repositories via the proxy server
• proxy-server contains source code for the proxy server (all generated by codegen except controllers)
  • swagger_server/controllers/repositories contains libraries to easily retrieve data from external repositories (all automatically generated)
• repositories is actually a swagger_server/controllers/repositories directory moved on top for convenience

How it works?

Normally, once user starts the harvester...

• sends 'repository' request to the proxy server and receives list of repositories supported by the proxy server
• sends 'ontology' requests to the proxy server in order to receive list of ontologies supported by each repository
  • proxy server requests data from repository
  • proxy server automatically (if there are 'pages' in response) iterates through the pages to gather all possible data
  • proxy server transform data in accordance with the mapping (provided in Meta/ontology object, see below)
  • send data in required format back to the harvester
• sends 'concepts' requests to the proxy server in order to receive list of terms/classes/concepts contained within each ontology in each repository
  • proxy server requests data from repository
  • proxy server automatically (if there are 'pages' in response) iterates through the pages to gather all possible data
  • proxy server transform data in accordance with the mapping (provided in Meta/concept object, see below)
  • send data in required format (it could be a jsonld) back to the harvester
• inserts data 'as-is' into mongodb collection: terms

Data

List of OpenAPI descriptors used for smart proxy project:

• smart_proxy.yaml describes API for the proxy server
• ebi_ac_uk_ols.yaml describes API for the EMBL-EBI Ontology Lookup Service (and includes mappings for data transfiguration)
• data_bioontology_org.yaml describes API for the NCBO BioPortal (incl. mappings for data transfiguration)
• data_agroportal_lirmm_fr.yaml describes API for the IBC AgroPortal (incl. mappings for data transfiguration)

Mapping

Note mapping for response data is included into OpenAPI descriptors, see Meta objects:

meta:
  type: "object"
  properties:
    ontology_xpath:
      type: "string"
      default: "embedded.ontologies"
    ontology:
      type: "object"
      properties:
        "@id":
          type: "string"
          default: "config.id"
        "dct:title":
          type: "string"
          default: "config.title"
        ...
    concept_xpath:
      type: "string"
      default: "embedded.terms"
    concept:
      type: "object"
      properties:
        "@id":
          type: "string"
          default: "iri"
        "skos:prefLabel":
          type: "string"
          default: "label"
        ...

where default property is a path to particular property within repository response (or path to array of ontologies/terms for _xpath properties).

Structure of properties is actually defined in OpenAPI descriptor for the proxy server:

definitions:
  repository:
    properties:
      - "@id"
      - "@type"
      - "dcat:landingPage"
      - "dct:title"
  ontology:
    properties:
      - "schema:includedInDataCatalog"
      - "@id"
      - "@type"
      - "dct:title"
      - "dct:description"
      - "omv:acronym"
      - "owl:versionInfo"
      - "dct:modified"
  concept:
    properties:
      - "rdfs:isDefinedBy"
      - "@id"
      - "skos:prefLabel"
      - "skos:definition"
      - "skos:note"
      - "skos:altLabel"

Experiments

Search across repositories

There is endpoint to request concept description from certain ontology within all registered repositories. If user sends request '/select?ontology={ontology}&iri={iri}', the proxy returns array of concepts (including its parents and children) in the following format:

{
  "query": "",
  "results": [
    {
      "children": [],
      "concept": {
        "@id": "",
        "http://www.w3.org/2000/01/rdf-schema#isDefinedBy": "",
        "http://www.w3.org/2004/02/skos/core#altLabel": [],
        "http://www.w3.org/2004/02/skos/core#definition": [],
        "http://www.w3.org/2004/02/skos/core#prefLabel": ""
      },
      "parents": [],
      "repository": ""
    },
  ]
}

Demo:

Server:

• get sample repositories: http://176.31.200.199/repositories
• get sample ontologies: http://176.31.200.199/repositories/ols/ontologies
• get sample concepts: http://176.31.200.199/repositories/ols/ontologies/aero/concepts
• search across repositories: http://176.31.200.199/search?query=ipomoea%20batatas
• get concept by IRI: http://176.31.200.199/select?ontology=GO&iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FGO_0043226
• get concept and its parent(s) and/or children: http://176.31.200.199/select?ontology=GO&iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FGO_0043226&fields=parents,children

Client:

• start processing: http://176.31.200.202/
• check results: see terms collection within proxy-db database (mongo)

Repositories:

The following repositories are supported:

• EMBL-EBI OLS
  • http://176.31.200.199/repositories/ols/ontologies
  • http://176.31.200.199/repositories/ols/ontologies/aero/concepts
• NCBO BioPortal (apikey=e7f84ef1-1530-4a02-85c6-6f907ab44ce8)
  • http://176.31.200.199/repositories/bioportal/ontologies
  • http://176.31.200.199/repositories/bioportal/ontologies/ICO/concepts
• IBC AgroPortal (apikey=587de5a7-22b9-4a9d-8c93-4bc4a40ebacf)
  • http://176.31.200.199/repositories/agroportal/ontologies
  • http://176.31.200.199/repositories/agroportal/ontologies/ICO/concepts

Technical data

All flask servers are located within /var/www/proxy-* directories and running 'behind' apache2 server. To add new repository, please generate it via swagger codegen, rename swagger-client (use any repository codename, eg.: ols, bioportal, agroportal) and put it into /var/www/proxy-server/swagger_server/controllers/repositories/ directory; then restart the server: /etc/init.d/apache2 restart