Remove Cloud Run deployer and associated CLI command

Contents (#580)

IMPORTANT: There is 1 breaking change.

Operations

• Add infrastructure needed to deploy example Cloud Run service

Testing

• Test latest deployed version of cloud run example service
• Test that app submodule log messages are handled

Reversions

• 💥 BREAKING CHANGE: Remove Cloud Run deployer and associated CLI command

---

Upgrade instructions

💥 Remove Cloud Run deployer and associated CLI command

Use terraform to create the Google Cloud infrastructure needed and the Octue GitHub actions and workflows to set up automatic deployment to Cloud Run. See here for an example workflow and here for an example terraform config.

Improve analysis logging

Summary

This PR refines the approach taken to adding metadata about an analysis to log messages emitted during its processing. It fixes an issue where developers couldn't test their apps based on the log messages emitted during an analysis run.

Contents (#578)

New features

• Enable Octue log handler by default

Enhancements

• Avoid using deprecated datafile property in runner
• Add default Octue service Dockerfile that uses python3.10

Fixes

• Swap out formatters instead of handlers during analyses
• Only use Octue formatter in analysis if envvar present

Refactoring

• Rename AnalysisLogHandlerSwitcher to AnalysisLogFormatterSwitcher
• Move run_logged_subprocess to octue.utils.processes

Testing

• Add tests for AnalysisLogFormatterSwitcher

Remove delivery acknowledgement timeout

Summary

This release removes the delivery acknowledgement timeout used when waiting for an answer from a child. The timeout had become mostly redundant since heartbeats were introduced and was causing difficult-to-debug errors. Heartbeats are now solely relied upon to determine whether a child is alive and responding to the parent. Delivery acknowledgement messages are still sent from the child to the parent, but their handling now only involves logging them.

Contents (#571)

IMPORTANT: There is 1 breaking change.

Fixes

• 💥 BREAKING CHANGE: Remove delivery acknowledgement timeout

---

Upgrade instructions

💥 Remove delivery acknowledgement timeout

Remove the delivery_acknowledgement_timeout argument from Child.ask and Service.wait_for_answer

Deprecate the Datafile.path property

Contents (#568)

Enhancements

• Deprecate the Datafile.path property

Fixes

• Ensure log records sent via Pub/Sub have a valid level number
• Handle empty expiration time in create-push-subscription CLI command
• Only log local file deletion in Runner if there are files to be deleted

Cache local metadata files and improve default push subscription creation behaviour

Summary

Add some performance and reliability improvements, including a local metadata file memory cache to speed up local metadata access for datafiles and datasets. Also set push subscriptions made via the CLI to never expire by default.

Contents (#567)

Enhancements

• Cache local metadata files' contents in memory to avoid wasted time repeatedly access the disk
• Break from datafile finding loop earlier if not in recursive mode

Fixes

• Set push subscriptions created via CLI to never expire by default

Refactoring

• Add and use local metadata context manager when updating local metadata files

Allow ignoring stored metadata when instantiating manifests

Contents (#563)

Enhancements

• Allow ignoring stored dataset and datafile metadata when instantiating manifests

Delete downloaded files after analysis is completed

Summary

After completion of an analysis, delete any files downloaded over its duration so the storage available to services isn't used up unnecessarily. This update frees service authors from having to deal with this themselves.

Contents (#562)

Fixes

• After completion of an analysis, delete any files downloaded over its duration

Testing

• Avoid test logging issue
• Move runner tests into correct test class

Improve delivery acknowledgement handling

Summary

Make a number of improvements and fixes to message handling when using pull subscriptions. Also update the small amount of testing that interacts with GCP to use a dedicated separate GCP project.

Contents (#558)

Enhancements

• Increase number of questions that can be asked concurrently in Child.ask_multiple to 32
• Make delivery acknowledgement and maximum hearbeat interval kwargs available in Child.ask
• Allow parents to start handling child responses from the first non-missed message (n + 1) if the first n were missed
• Add question UUID to heartbeat log messages
• Improve PushSubscriptionCannotBePulled error message

Fixes

• Mark question as delivered on receipt of first response from child in case the delivery acknowledgement message is missed
• Stop loss of delivered question UUIDs if local metadata file does not yet exist
• Avoid message gap greater than the delivery acknowledgement timeout causing failure to receive child messages
• Allow a start time of zero in message handler

Dependencies

• Update to latest versions of protobuf and werkzeug to avoid security issues

Operations

• Add terraform configuration for new test project

Refactoring

• Simplify nested conditional
• Minimise code within try/except block in OrderedMessageHandler
• Move message recording into OrderedMessageHandler._handle_message
• Factor out raising message handling error in message handler
• Rename OrderedMessageHandler.received_messages to handled_messages

Testing

• Use new URI for Strands JSON schemas in tests
• Use dedicated GCP project for testing services

Improve deployment dockerfiles

Summary

Improve the Cloud Run and Dataflow deployment dockerfiles and enable poetry-based packages on Dataflow.

Contents (#554)

Enhancements

• Use windpioneers/gdal-python as base in Cloud Run default deployment Dockerfile
• Use latest apache-beam base for Dataflow default deployment dockerfile
• In both the Cloud Run and Dataflow default deployment dockerfiles:
  • Improve dependency caching
  • Remove unnecessary installation of curl
  • Only install from one type of dependency file while building
  • Use /workspace as WORKDIR in Dockerfile
  • Use /root/.poetry for POETRY_HOME

Fixes

• Always install poetry in deployment dockerfiles

Operations

• Use latest octue/check-semantic-version GitHub action

Add code scanning

Summary

Enable automated code security scanning via GitHub's CodeQL.

Contents (#553)

Operations

• Add codeql.yml workflow to enable code scanning