Skip to content

Latest commit

 

History

History
489 lines (372 loc) · 19.8 KB

release-1.2.md

File metadata and controls

489 lines (372 loc) · 19.8 KB
Raw

EOEPCA System - Release 1.2

Release 1.2 includes versions of the following building blocks...

Component Chart Application
Login Service 1.2.1 v1.1.2
User Profile 1.1.6 v1.1
Resource Guard
-> PEP (Policy Enforcement Point)
-> UMA User Agent		 1.2.0
1.1.5
1.2.0		 u1.2.0:p1.1.5
v1.1
v1.2.0
PDP (Policy Decision Point) 1.1.6 v1.1
ADES (Application Deployment & Execution Service) 2.0.4 2.0.8
PDE (Processor Development Environment) 1.1.12 2.0.2
Resource Catalogue 1.2.0 3.0.0
Data Access 1.2.5 2.2.8
Workspace 1.2.0 1.2.0

See the Release v1.2 Description for more details.

Changes Since v1.1

The release includes bugfixes to the v1.1 building blocks, plus the following main enhancements...

Component Feature
ADES ades v2 aligned to OGC API Processes 1.0 standard
ADES improved documentation for stage-in/out - see https://github.com/EOEPCA/proc-ades/wiki/Stagein%20Stageout%20Interfaces
ADES improvements to execution error reporting
ADES support for CWL parsing of multi-type inputs
ADES fix for 'too long' workflow ID
ADES fix for undeploy
ADES fix stage-in from S3
ADES fix stage-out to workspace
Resource Catalogue support for Application Packages (of type application), in accordance with the OGC Best Practise for APs
Resource Catalogue registration of Collections as resources
Resource Catalogue registration of ADES instances as resources (of type service)
Resource Catalogue improvements to resultset ATOM representation
Data Access improvements to the renderer for performance and resilience
Data Access performance improvements to the View Server web client
Data Access add seeder service for cache pre-population
Data Access fix registration of processing results to user workspace
Data Access fix WCS rendering
"Data Holding" support for Landsat-8 data harvested from CREODIAS
Workspace API use of templates to reduce sub-chart dependencies
Workspace API remove flux pre-requisite
Workspace API registration of Application Packages, Collections and ADES
Login Service add Registration page
Login Service support for integration of customised GUI look-and-feel
Login Service support for custom-specified Kubernetes namespace
PDP support for custom-specified Kubernetes namespace
PDP support for TLS endpoint protection
User Profile support for custom-specified Kubernetes namespace
User Profile support for TLS endpoint protection

NOTE that this release includes major release 2.0 of the ADES. Significantly, ADES 2.0 has an updated REST interface that now conforms to the OGC API Processes 1.0 standard - whereas the ADES 1.0 implemented a draft of this standard from OGC Testbed-14. See the ADES wiki pages for deploy and execute to understand the differences.

Known Issues

The following issues are known to affect the v1.2 release.

  • None

Release 1.2 Scope

The release demonstrates the following capabilities:

  • User authentication:
    • Login with GitHub
    • Login with ESA Commercial Operator Identity Hub (COIH)
    • Login with username/password
  • Client Registration
    • Dynamic client registration via SCIM endpoint
  • Authorisation
    • Dynamic Resource Registration
      Resource servers dynamic registration of resources
    • Resource protection
      Enforcing a policy in which resources are owned and protected accordingly
    • Policy-based resource protection
      Enforcing policy based upon policy rules maintained in the PDP
  • Processing Capabilities (ADES resource server)
    • OGC API Processes 1.0
    • OGC WPS 2.0
    • Secure protected resource server, with access policy enforcement via PEP
    • List available processes
    • Deploy process (docker container with CWL application package)
    • Execute process (create job)
    • Get job status
    • Data stage-in via STAC/OpenSearch catalogue reference
    • Data stage-out to S3 bucket
    • Undeploy process
    • Integration of Calrissian CWL Workflow engine
      Provides native Kubernetes integration and out-of-the-box support for a variety of execution patterns - such fan-in, fan-out, etc.
    • Dedicated user 'context' within ADES service
  • Processor Development Environment (PDE)
    • JupyterHub for multi-user sessions
    • JupyterHub integrated with Login Service for user authentication
    • JupyterLab for user PDE instance
    • Jupyter Notebooks for interactive analysis
    • Theia IDE to develop using an integrated development environment
    • Tools for application package testing
  • Resource Catalogue
    • Implements ISO Metadata Application Profile 1.0.0
    • Support for ISO-19115-1 and ISO-19115-2
    • OGC CSW 3.0.0 and 2.0.2 interfaces
      • Certified OGC Compliant and OGC Reference Implementation for both CSW 2.0.2 and CSW 3.0.0
      • Harvesting support for WMS, WFS, WCS, WPS, WAF, CSW, SOS
      • Federated catalogue distributed searching
    • OGC API Records
    • SpatioTemporal Asset Catalog (STAC) 1.0.0-rc1
    • OpenSearch
      • OGC OpenSearch Geo and Time Extensions
      • OGC OpenSearch EO Extensions
  • Data Access Service
    • OGC WMS 1.1 - 1.3 interfaces
    • OGC WMTS 1.0 interfaces with automatic caching
    • OGC WCS 2.0 interfaces with EO Application Profile
    • OGC OpenSearch with EO, Geo and Time Extensions
    • Workspace management API
    • Dataset registration API
    • Registration schemes for Sentinel-2 L1C/L2A data in Data Access Service and Ressource Catalogue
  • Data Harvesting
    • Population of resource catalogue and data-access services from external data offerings
    • Sources supported include: file-system, search service (e.g. OpenSearch), catalog file
    • Filters, e.g. time, bbox, collection
    • Post-processing to adjust harvested results, e.g. for completion of missing metadata
    • Harvested results are transformed to STAC items for registration
  • End-to-end Processing Execution
    • Authenticated user accessing protected ADES endpoints
    • Dynamic creation of ADES user context with dynamic resource protection
    • Processing inputs discovered in Resource Catalogue
    • Processing inputs accessed via S3 (e.g. CREODIAS eodata)
    • Processing stage-in using STAC file to describe inputs
    • Processing execution on ADES
    • Processing stage-in using STAC file to describe inputs
    • Processing stage-out to S3 bucket
    • Processing stage-out using STAC file to describe outputs
    • Secure interfacing between ADES and user's protected Workspace
      ADES client -> Get Workspace Details / (De-)Register Resources
  • User Workspace
    • Secure protection of user-owned resources, with access policy enforcement via PEP
      • User-specific S3 bucket for resource storage
      • User-specific Resource Catalogue
      • User-specific Data Access Services
    • Secure protected management interface (workspace-api), with access policy enforcement via PEP
    • Management functions via REST API...
      • Create workspace (admin)
      • Get workspace details (user)
      • Delete workspace (admin)
      • Patch workspace (admin)
      • Redeploy workspace (admin)
      • Register resources (user)
      • Deregister resources (user)
  • Sample application: s-expression for EO product band math
    Three application packages based-upon s-expression:
    • App s-expression
    • App Water Mask
    • App NVDI
  • Sample application: Normalized Hotspot Indices

Building Blocks

This section identifies the version of the building blocks components comprising this release, and provides links for further information. For each, in this repository, we include an 'Example' deployment configuration using a flux HelmRelease resource - these must be adapted for individual deployments.

Alternatvely, for deployment advice, see the Deployment Guide which provides full system deployment descriptions, examples and supporting scripts.

User Management

Login Service

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • Gluu Server:
    • gluufederation/config-init:4.1.1_02
    • gluufederation/oxauth:4.1.1_03
    • gluufederation/oxtrust:4.1.1_02
    • gluufederation/wrends:4.1.1_01

Resource Guard

The Resource Guard acts as an umbrella for the protection of resource servers via the pep-engine (Policy Enforcement Point) and uma-user-agent (UMA User Agent) components.

Resource Guard:

Policy Enforcement Point (PEP)

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • mongo (latest)

UMA User Agent

Resources

Resources to support deployment and configuration...

Containers

Policy Decision Point (PDP)

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • mongo (latest)

User Profile

Resources

Resources to support deployment and configuration...

Containers

Processing and Chaining

ADES

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • Stage-in: terradue/stars:2.9.2
  • Stage-out: terradue/stars:1.0.0-beta.11

Processor Development Environment (PDE)

Containers

Sample Applications

Sample application packages for deployment and execution on the ADES:

Resource Management

Resource Catalogue

Resources

Resources to support deployment and configuration...

Containers

  • pycsw (version eoepca-1.2.0)
    • Image: geopython/pycsw:eoepca-1.2.0

Additional container images:

  • Database: postgis/postgis:12-3.1

Data Access Services

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • Cache: registry.gitlab.eox.at/vs/cache:release-2.0.24
  • Client: registry.gitlab.eox.at/vs/client:release-2.0.33
  • Database: bitnami/postgresql:11.13.0-debian-10-r40
  • Redis: bitnami/redis:6.0.8-debian-10-r0
  • Scheduler: registry.gitlab.eox.at/vs/scheduler:release-2.0.2
  • Seeder: registry.gitlab.eox.at/vs/cache/seeder:release-2.0.24

Workspace

Resources

Resources to support deployment and configuration...

Containers

Bucket Operator

Resources

Resources to support deployment and configuration...

Containers