Persistence Hooks for Storage Path Management

This commit will allow users to share declared volumes and delete volume
paths exposed by the docker manifest with the help of Persistence Hooks.
The volume paths are processed according to the javascript functions
and returned to PREvant in an array of array, where each element of the
array is pair of service name and volume path. This will then be passed
to the infrastructure for storage creation.

The hooks also allows adding of any additional volume paths that are
undeclared in the image.

api/README.md

@@ -200,11 +200,11 @@ storageStrategy = 'mount-declared-image-volumes'
 - `none` (_default_): Companion is deployed without persistent storage.
 - `mount-declared-image-volumes`: Mounts the volume paths declared within the image, providing persistent storage for the companion.
 
-## Hooks
+## Deployment Hooks
 
-Hooks can be used to manipulate the deployment before handing it over to actual infrastructure and they are able to manipulate all service configurations once for any deployment REST API call. For example, based on the deployment's app name you can decide to reconfigure your services to use a different DBMS so that you are able to verify that your services work with different DBMSs.
+Deployment Hooks can be used to manipulate the deployment before handing it over to actual infrastructure and they are able to manipulate all service configurations once for any deployment REST API call. For example, based on the deployment's app name you can decide to reconfigure your services to use a different DBMS so that you are able to verify that your services work with different DBMSs.
 
-Technically, hooks are Javascript files that provide functions to modify all service configurations of a deployment. For example, add following section to your PREvant configuration. This configuration snippet enables the _deployemnt hook_ that will be used to modify the services' configurations.
+Technically, hooks are Javascript files that provide functions to modify all service configurations of a deployment. For example, add following section to your PREvant configuration. This configuration snippet enables the _deployment hook_ that will be used to modify the services' configurations.
 
 ```toml
 [hooks]
@@ -233,6 +233,34 @@ PREvant calls this function with the app name (see `appName`) and an array of se
 | `env`         | A map of key and value containing the environment variables that will be used when creating the container. |
 | `files`       | A map of key and value containing the files that will be mounted into the container.                       |
 
+## Persistence Hooks
+Persistence Hooks in PREvant are designed to handle data persistence by managing volume paths of services. For example, you can use a Persistence Hook to configure any services that share a persistence path or add missing additional paths that caters to unreleased features.
+
+Like Deployment Hooks, these hooks are Javascript files that provide functions to manage and manipulate persistence settings. This configuration snippet enables the _persistence hook_ that will be used to modify the services' configurations.
+
+```toml
+[hooks]
+persistence = 'path/to/persistenceHook.js'
+```
+
+The hook at `path/to/hook.js` must provide following Javascript function:
+
+```javascript
+function persistenceHook(appName, serviceConfigs) {
+  //Return Array of an Array of pairs of service name and path.
+}
+```
+Note: The input to the Persistence Hook is an array of name and path pairs. If multiple paths are declared in the configuration, they are split and treated as separate pairs.
+
+PREvant will invoke this function with an array of service config with name and path as fields. This input can be modified according to the required persistence logic and must be returned by the function in an Array of Arrays format. The output structure allows for grouping shared paths or treating them as individual, based on the deployment's specific requirements.
+
+The fields in each object of the serviceConfig are:
+
+| Key           | Description                                                                                                |
+|---------------|------------------------------------------------------------------------------------------------------------|
+| `name`        | The service name (readonly).                                                                               |
+| `path`       | The path declared by the docker manifest of the image file.              |
+
 ## Registries
 
 Private registries require login information, therefore, PREvant offers authentication for secured registries. Add following block to your configuration file:

api/src/apps/mod.rs

@@ -280,7 +280,7 @@ impl AppsService {
             .extend_with_templating_only_service_configs(configs_for_templating)
             .resolve_image_manifest(&self.config)
             .await?
-            .apply_templating()?
+            .apply_templating(&self.config)?
             .apply_hooks(&self.config)
             .await?;
 
@@ -398,6 +398,8 @@ pub enum AppsServiceError {
     UnableToResolveImage { error: ImagesServiceError },
     #[fail(display = "Invalid deployment hook.")]
     InvalidDeploymentHook,
+    #[fail(display = "Invalid persistence hook.")]
+    InvalidPersistenceHook,
 }
 
 impl From<ConfigError> for AppsServiceError {

api/src/apps/routes.rs

@@ -360,7 +360,8 @@ impl From<AppsError> for HttpApiError {
             | AppsError::InvalidServerConfiguration { .. }
             | AppsError::InvalidTemplateFormat { .. }
             | AppsError::UnableToResolveImage { .. }
-            | AppsError::InvalidDeploymentHook => {
+            | AppsError::InvalidDeploymentHook
+            | AppsError::InvalidPersistenceHook => {
                 error!("Internal server error: {}", error);
                 StatusCode::INTERNAL_SERVER_ERROR
             }

api/src/deployment/deployment_unit.rs

@@ -25,12 +25,13 @@
  */
 use crate::apps::AppsServiceError;
 use crate::config::{Config, StorageStrategy};
-use crate::deployment::hooks::Hooks;
 use crate::infrastructure::TraefikIngressRoute;
 use crate::models::{AppName, ContainerType, Image, ServiceConfig};
 use crate::registry::{ImageInfo, ImagesService, ImagesServiceError};
 use std::collections::{HashMap, HashSet};
 
+use super::hooks::Hooks;
+
 pub struct Initialized {
     app_name: AppName,
     configs: Vec<ServiceConfig>,
@@ -92,11 +93,13 @@ pub struct WithAppliedTemplating {
 pub struct WithAppliedHooks {
     app_name: AppName,
     services: Vec<DeployableService>,
+    volume_groups: Vec<Vec<ServicePath>>,
 }
 
 pub struct WithAppliedIngressRoute {
     app_name: AppName,
     services: Vec<DeployableService>,
+    volume_groups: Vec<Vec<ServicePath>>,
 }
 
 pub struct DeploymentUnitBuilder<Stage> {
@@ -106,6 +109,7 @@ pub struct DeploymentUnitBuilder<Stage> {
 pub struct DeploymentUnit {
     app_name: AppName,
     services: Vec<DeployableService>,
+    volume_groups: Vec<Vec<ServicePath>>,
 }
 
 #[derive(Clone, Debug)]
@@ -123,6 +127,27 @@ pub struct DeployableService {
     declared_volumes: Vec<String>,
 }
 
+#[derive(Debug, Clone, Serialize, Deserialize, Hash, PartialEq, Eq)]
+#[serde(rename_all = "camelCase")]
+pub struct ServicePath {
+    service_name: String,
+    path: String,
+}
+
+impl ServicePath {
+    pub fn new(service_name: String, path: String) -> ServicePath {
+        ServicePath { service_name, path }
+    }
+
+    pub fn service_name(&self) -> &String {
+        &self.service_name
+    }
+
+    pub fn path(&self) -> &String {
+        &self.path
+    }
+}
+
 impl DeployableService {
     #[cfg(test)]
     pub fn new(
@@ -174,6 +199,10 @@ impl DeploymentUnit {
     pub fn app_name(&self) -> &AppName {
         &self.app_name
     }
+
+    pub fn volume_groups(&self) -> &Vec<Vec<ServicePath>> {
+        &self.volume_groups
+    }
 }
 
 impl DeploymentUnitBuilder<Initialized> {
@@ -314,11 +343,24 @@ impl DeploymentUnitBuilder<WithTemplatedConfigs> {
 impl DeploymentUnitBuilder<WithResolvedImages> {
     pub fn apply_templating(
         self,
+        app_config: &Config,
     ) -> Result<DeploymentUnitBuilder<WithAppliedTemplating>, AppsServiceError> {
         let mut services = HashMap::new();
+        let image_infos = &self.stage.image_infos;
 
         for config in self.stage.configs.iter() {
             let templated_config = config.apply_templating(&self.stage.app_name)?;
+            let declared_volumes = match app_config.hook("persistence") {
+                Some(_) => match image_infos.get(config.image()) {
+                    Some(image) => image
+                        .declared_volumes()
+                        .iter()
+                        .map(|path| path.to_string())
+                        .collect(),
+                    None => Vec::new(),
+                },
+                None => Vec::new(),
+            };
 
             services.insert(
                 config.service_name().clone(),
@@ -329,7 +371,7 @@ impl DeploymentUnitBuilder<WithResolvedImages> {
                         &self.stage.app_name,
                         config.service_name(),
                     ),
-                    declared_volumes: Vec::new(),
+                    declared_volumes,
                 },
             );
         }
@@ -379,8 +421,6 @@ impl DeploymentUnitBuilder<WithResolvedImages> {
                 .merge_with(&companion.templated_companion);
         }
 
-        let image_infos = &self.stage.image_infos;
-
         // Exclude service_companions that are included in the request
         services.extend(
             service_companions_of_config
@@ -474,10 +514,9 @@ impl DeploymentUnitBuilder<WithResolvedImages> {
 
         let declared_volumes = match storage_strategy {
             StorageStrategy::NoMountVolumes => Vec::new(),
-            StorageStrategy::MountDeclaredImageVolumes => volume_paths
-                .into_iter()
-                .map(|path| path.to_owned())
-                .collect(),
+            StorageStrategy::MountDeclaredImageVolumes => {
+                volume_paths.into_iter().cloned().collect()
+            }
         };
 
         match strategy {
@@ -533,11 +572,15 @@ impl DeploymentUnitBuilder<WithAppliedTemplating> {
         let services = hooks
             .apply_deployment_hook(&self.stage.app_name, self.stage.services)
             .await?;
+        let volume_groups = hooks
+            .apply_persistence_hooks(&self.stage.app_name, &services)
+            .await?;
 
         Ok(DeploymentUnitBuilder {
             stage: WithAppliedHooks {
                 app_name: self.stage.app_name,
                 services,
+                volume_groups,
             },
         })
     }
@@ -557,6 +600,7 @@ impl DeploymentUnitBuilder<WithAppliedHooks> {
             stage: WithAppliedIngressRoute {
                 app_name: self.stage.app_name,
                 services: self.stage.services,
+                volume_groups: self.stage.volume_groups,
             },
         }
     }
@@ -565,6 +609,7 @@ impl DeploymentUnitBuilder<WithAppliedHooks> {
         DeploymentUnit {
             app_name: self.stage.app_name,
             services: self.stage.services,
+            volume_groups: self.stage.volume_groups,
         }
     }
 }
@@ -574,6 +619,7 @@ impl DeploymentUnitBuilder<WithAppliedIngressRoute> {
         DeploymentUnit {
             app_name: self.stage.app_name,
             services: self.stage.services,
+            volume_groups: self.stage.volume_groups,
         }
     }
 }
@@ -630,7 +676,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -669,7 +715,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -725,7 +771,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -773,7 +819,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -818,7 +864,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -854,7 +900,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -894,7 +940,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -949,7 +995,7 @@ mod tests {
             .extend_with_templating_only_service_configs(vec![sc!("postgres", "postgres:alpine")])
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -1004,7 +1050,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -1057,7 +1103,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -1109,7 +1155,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -1155,7 +1201,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .build();
@@ -1186,7 +1232,7 @@ mod tests {
             .extend_with_templating_only_service_configs(Vec::new())
             .resolve_image_manifest(&config)
             .await?
-            .apply_templating()?
+            .apply_templating(&config)?
             .apply_hooks(&config)
             .await?
             .apply_base_traefik_ingress_route(TraefikIngressRoute::with_rule(