Apex

This is Apex, a middleware implementation for SAPIENT.

Overview

Apex has three main functions:

• Routing messages between the components: For example, a task from the Fusion Node (FN) for sensor 7 will only go to the relevant Edge Node's (EN) connection. If the sensors connect directly to the FN, rather than via Apex, then all tasks would get broadcast to all connections.
• Validating messages (optional): Validation against the Protobuf schema of the SAPIENT standard (BSI Flex 335 v1.0, BSI Flex 335 v2.0 & so on.)
• Archiving the messages: Messages are recorded to a database (a SQLite file), allowing for review of what happened afterwards, including the possibility of being replayed in the FN.

A variety of smaller tasks are also handled by Apex. For example, registration messages and status reports are cached so that if the FN connects after an EN then the FN can still be sent the earlier relevant messages. Some validation or filtering of messages is also possible e.g. filtering out detections with a low detection confidence.

Apex Releases & Installation Types.

Apex releases are made available in multiple forms, depending on the use case.

1. Apex Binaries
   Meant for end-users.
   These are compiled Apex executables, ready to use.
   Does not need an internet connection.

2. Apex Sources + Python Virtual Environment
   Meant for technical users.
   Contains the Apex (python) source code.
   Included is a pre-configured python virtual environment with all required packages already installed.
   Does not need an internet connection.
   The virtual enviornment should be activated.
   A working python installation is recommended.

3. Apex Sources
   Meant for technical users or developers
   Contains the Apex (python) source code.
   Needs a working internet connection.
   Needs a working python, poetry installation.
   See Setup for Development below.

Optional Components

Elasticsearch Setup

This is required for Apex's REST API to work.

Installation of Elasticsearch On Windows

1. Install Elasticsearch

   • Open a console in the root of the apex directory.
   • Run: .\install_elastic.bat .
     • The install_elastic.bat takes a single argument the . above, indicates that elastic will be install in the current directory
     • This will download and unpack an elasticsearch binary to the root of the Apex folder.
     • Alternatively, follow the steps on the official Elasticsearch website to install elasticsearch.

2. Execute Elasticsearch

   • Open a console in the root of the apex directory.

   • Run: start_elastic.bat .

     • The start_elastic.bat takes a single argument, the location of the installed elastic. Adjust the argument as needed, if you already have an elastic instance in another directory
     • The first time you execute Elasticsearch, a password is generated for the elastic user (look for the console output that states Password for the elastic user (reset with "bin/elasticsearch-reset-password -u elastic"): in the window that you have run Elasticsearch in).
       Copy this password into the elasticConfig.password option in apex_config.json

   • Set the elasticConfig.enabled flag in apex_config.json to true

   • After Elasticsearch has started, update the elasticConfig.certLocation option in apex_config.json so that it is the full path to the http certificate that was generated by Elasticsearch (file is elasticsearch-8.11.1\config\certs\http_ca.crt)

Elasticsearch With Docker

Requires a Docker installation on the host PC.
There are docker files in the tests directory, which allow running up a Elasticsearch instance in Docker.

Run the following:

cd tests/
docker compose up

This will start a docker compose instance of elastic search and of Kibana. Setting up both will take a minute. The credentials for elastic search are generated automatically and stored in tests/credentials. Users can log in to Kibana using the username, password, and port specified in tests/.env.

Setup for Development

1. Install Python Version >= 3.9

2. Install Poetry globally by running pip install poetry

3. Create a python virtual environment manually in a console window. Example on windows: python -m venv venv and venv\scripts\activate.bat

4. Run poetry install --all-extras to install packages & dependencies (including gui and dev dependencies) and the current apex package version in editable mode

   Note: It is possible, though not recommended, to install dependencies with pip install -r requirements.txt. This will not install the apex as a package properly.

5. Optionally, only if apex is in a git repo, run pre-commit install so that git runs linting/formatting before each commit, as well as keep the repo consistent (generated proto bindings, poetry lock file, etc...). The linting and formatting in this step can also performed in a CI/CD system automatically.

6. Optionally, when changing proto files in sapient_msg, run poetry run python run_protoc.py to regenerate the python bindings. Alternatively, with 5 above, run pre-commit run protobuf --all, or simply commit the changed proto files.

Dependency and version management

It is always possible to edit pyproject.toml manually and then run poetry lock (there are pre-commit hooks to ensure consistency).

• adding development dependencies: poetry add -G dev dependency-name
• adding true apex dependencies: poetry add dependency-name or poetry add -G gui dependency-name
• bump up the version: poetry version patch (or minor, major, etc...)
• build wheels and distribution files: poetry build

The protobuf definitions for the Sapient protocols can be found in sapient_msg. See here for more information regarding adding new versions.

How to run Apex directly

Elasticsearch (Optional)

• When starting Elasticsearch wait for a log with the message
  Cluster health status changed from [RED] to [YELLOW/GREEN]
  before starting Apex to allow Elasticsearch to start fully

From Binaries

• Optionally start elastic service: run .\start_elastic.bat
• Find & Execute run_all.bat which will launch apex.exe and apex_gui.exe
• or execute apex.exe and apex_gui.exe from the bin subdirectory.
• There maybe Windows UAC/Firewall/Defender prompts, which need to be accepted & network access allowed, for Apex to run.

From Source

• Optionally start elastic service with .\start_elastic.bat or cd tests; docker compose up
• Activate the virtual environment with venv\scripts\activate.bat
• Execute Apex with
  • apex or
  • python sapient_apex_server\apex.py
• Execute Apex GUI with
  • apex_gui or
  • python sapient_apex_gui\apex_gui.py

Apex REST API

• When Apex starts, it also starts a REST Server, which provides the Apex REST API.
• Default address for the Apex REST Server is http://127.0.0.1:8080
• Swagger/openapi/web interface address is http://127.0.0.1:8080/docs
  • This can be used to see the integrated documentation for all the REST endpoints, along with their parameters, schema & error codes.
  • REST endpoints can be executed using the swagger webui.
• The REST API requires that the elasticsearch is installed & configured correctly.
  • Ensure elasticConfig.enabled flag in apex_config.json to true
  • Ensure that apex is able to connect to the elastic instance successfully.
  • Otherwise the HTTP error code 503 Database Service Unavailable or not started or Internal Error will be returned for all REST Endpoints.
• Apex REST Server configuration is done by apiConfig in apex_config.json

How to build Apex as a self-contained executable

• To create the exe(s), activate your virtual environment and run deploy\build_nuitka.bat and wait a while!
• This will create apex.exe, apex_gui.exe & replay.exe in the deploy\bin subdirectory.
• You can easily run both of these using deploy\run_all.bat
• The build script also zips the binary deployment files into an archive: apex.zip
• To deploy to another machine, simply copy the generated apex.zip and extract

How to build the Apex Deliverables

• Apex deliverables are in multiple formats.
• Use make_apex_delivery.bat which takes one argument, the apex zip containing the sources.
• Typically this is the downloaded apex source zip file from gitlab for a particular git tag.
• This script will unzip the sources, create the python virtual enviornment & build apex binaries and finally zip all them up into separate deliveries.
• Run it out of source to avoid creating files in the apex repo, example from a parent directory:
  apex\make_apex_delivery.bat apex-v4.0.0.zip

Apex Configuration

Configuration is done by editing the apex_config.json
Explanation of the fields:

{
  // INFO for normal trace level, or DEBUG for extra trace
  "logLevel": "INFO",

  // List of connections
  "connections": [
    {
      // Child (ASM), Peer (DMM), Parent or Recorder
      "type": "Child",

      // Whether to connect outbound (true) or listen for incoming connections (false)
      "outbound": true,

      // Port to connect to or listen on
      "port": 5020,

      // Host to connect to (only if "outbound" is true)
      "host": "127.0.0.1",

      // Parent connection only, receive all messages, not just high-level messages
      "forwardAll": false,

      // Sapient protocol used by the node. If absent,then it defaults to
      // "VERSION 6" if the encoding is XML, and to the latest standard if the
      // encoding is PROTO.
      // Options include: "VERSION 6", "BSI Flex 335 v1.0", "BSI Flex 335 v2.0"
      "icd_version": "BSI Flex 335 v2.0"

      // Output format. Currently either XML or PROTO. XML is only compatible
      // with VERSION 6 (and vice-versa).
      "format": "PROTO"
    },
    {
      // ... more connections go here ...
    }
  ],

  // Elastic Server configuration. Needed for REST API
  "elasticConfig": {
    "enabled": false,
    "host": "localhost",
    "port": 9200,
    "useSsl": true,
    // Location of the elastic certificates
    "certLocation": "C:\\elasticsearch-8.11.1\\config\\certs\\http_ca.crt",
    "user": "elastic",
    "password": "password"
  },
  // Apex REST Server configuration. Needs Elasticsearch to be installed & enabled.
  "apiConfig": {
    "host": "127.0.0.1",
    "port": 8080
  }

  // Enables a adjustment to deal with differences in clock sync
  "enableTimeSyncAdjustment": false,

  // Maximum allowed buffered message size; helps detects callers forgetting to null terminate
  "messageMaxSizeKb": 1024,

  // Ignores detections below a confidence threshold (introduced for a particular trial)
  "detectionConfidenceFiltering": {
    "enable": false,
    "threshold": 0.5,
    "storeInDatabase": true
  },

  // Configures the node_id used in registrationAck and error messages sent from the middleware
  "middlewareId": "5913c0f4-9f89-4c01-ab90-939099797c4f",

  // Allows a registration message to be sent from the DMM.
  "allowPeerRegistration": true,

  // BSI Flex 335 v2.0 Related
  // Send RegistrationAck for Node registrations.
  // true = Middleware sends it. false implies the FusionNode/DMM will send it.
  "sendRegistrationAck": true

  // Configuration of the database rollover
  "rollover": {
    "enable": true,
    "unit": "days",
    "value": 1
  }

  "validationOptions": {
    // Validation types to enable
    "validationTypes": [
      // All mandatory fields are present
      "mandatory_fields_present",

      // All mandatory Oneof has at least one of its elements present
      "mandatory_oneof_present",

      // All mandatory repeated has at least one element present
      "mandatory_repeated_present",

      // No unknown fields have been included
      "no_unknown_fields",

      // No unknown numeric enum values
      "no_unknown_enum_values",

      // UUID and ULID fields are in their correct respective formats
      "id_format_valid",

      // Message timestamp within reasonable time from current time
      "message_timestamp_reasonable",

      // Detection timestamp with reasonable time between detection and current time
      "detection_timestamp_reasonable",

      // Validate if the registration message ICD version is within the supported list
      "supported_icd_version"
    ],

    // If true (and id_format_valid is present) then UUID must be UID v4, lower case,
    // with correct dashes, and no braces or URN prefix; ULID must be upper case
    "strictIdFormat": true,

    // Minimum and maximum relative times for message timestamp validation
    "messageTimestampRange": [-0.9, 0.1],

    // Minimum time between detection messages
    "detectionMinimumGap": 0.08,

    // List of supported ICD versions
    "supportedIcdVersions": ["BSI Flex 335 v1.0", "BSI Flex 335 v2.0"]
  }
}

License

Under the terms of the contract that this software was developed, the contractor retains the Intellectual Property Rights (IPR) to the software, but has granted Dstl a permissive license to re-license the software as Open-Source Software. Dstl has chosen to release this software with the Apache 2.0 license.

Except where noted otherwise, the Apex SAPIENT Middleware software is licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.