Release Notes of ProActive V13.0.0

[mboussaa]

ActiveEon is constantly adding new capabilities, so you can leverage the latest technologies to experiment and innovate more quickly.

Version V13.0.0 “Gravity” - September 2022

Discover an overview of the new ActiveEon features, expansions, and hotfixes as released in the latest release: V13.0.0.

ProActive New Features

ProActive Workflow and Scheduling

• ProActive installation within Kubernetes cluster

• The ProActive instance can be easily deployed within a Kubernetes cluster environment.

• Administrators can run ProActive instances in containers within a Kubernetes cluster and apply different ProActive configuration parameters such as different databases, users, and passwords.

• This function enables the customer to take full advantage of Kubernetes capabilities in terms of high availability, disaster recovery, and full portability across platforms.

• A single ProActive instance can serve multiple groups of users

• Administrators can now create multi-tenants within the same instance of ProActive making the solution more convenient, future-proof, and scalable for large projects across the company.

• Each tenant can leverage the available computing resources within the organization while keeping data and workloads completely separate from other teams.

• Better usage of resources: By sharing machines among multiple users and using the same infrastructure, the organization can optimize available computing resources.

• Lower costs: Thanks to the centralized nature of the orchestrator, each tenant can benefit from centrally managed infrastructure and software, allowing the IT department to offer its services at a much lower price due to lower operational costs.

• Streamlining release: Instead of installing new versions separately on each teams’ servers, the multi-tenant package only needs to be installed on a single server.

• Significant increase in task processing

• Our new capability enables customers to deliver automation much faster.

• The R&D team has delivered an improved version of the workload scheduler, doubling the number of tasks processed by seconds.

• By upgrading to the new release, your ProActive solution will now process the workload twice as fast as before and can process more than 2 million tasks per day.

• Run dynamic workflows that adjust their behavior based on parameter values

• ActiveEon provides a great amount of flexibility with respect to how you want to trigger your workflow. With ActiveEon’s new “Action/Signal” function, administrators can dynamically overwrite parameter values for a given run without having to redeploy their workflow.

• It’s often convenient to have a workflow that is capable of handling or responding to different inputs. For example, a workflow might represent a series of steps that could be repeated for information coming from different APIs, databases, or credentials — all of which reuse the same processing logic. Alternatively, you might want to use an input parameter that we call “Action/Signal” to affect the workflow processing itself.

• The Action/Signal function is a powerful way to act upon a workflow in real-time during its process.
  Some examples of use cases are:

• An agent can pause the workflows and change the input parameters in real-time. (Changing the compute target, for example)

• A third-party software using REST can alter the workflow inputs during the execution of the workflow.

• The workflow can pause and request an agent to select different options: such as where to archive the data: 1- in your favorite web service 2- in the on-premises storage for confidentiality requirements.

Below is a screenshot where users can select to re-run the ML model by changing some parameters:

Changing input dynamically for workflow parameters enables the process to be more flexible and robust, providing a more refined user experience.

• Advanced functions for monitoring workloads
  Administrators will benefit from a full set of new features added to the Automation Dashboard – please see below the top five functions.

Feature 1:

• Launching third-party services directly from the dashboard:
  Users can launch third-party software directly from the ActiveEon dashboard. This is particularly useful when users need information from another application and access it at a press of a button.
  

Two customers use cases:

• An Administrator/Monitoring person could dynamically open the company’s centralized database describing the input parameters for each workflow template and launch a new workflow with the relevant input parameters.

• Data Scientists may want to open their notebooks in Jupyter Lab and directly launch the model from Jupyter Lab into ActiveEon (using Pragma which will be described later in this document). We will discuss later on how Data scientists who have launched their experimentations can also dynamically launch Tensor Board without leaving the ActiveEon interface to visualize the progress of the model training or experiment.
  

• The service life cycle is entirely controllable by the administrator, like any other job. They can pause, run, or stop the service directly from the dashboard.

• This feature demonstrates the power of ActiveEon’s open architecture, where any service can be launched from ProActive in real-time and reciprocally third-party solutions can call any ProActive service through REST API.

Feature 2:

• Launching a third-party software during a workflow execution:
  Very similar to feature 1 but even more powerful.
  Administrators/Users can launch a third-party solution in real-time during the execution of a workflow. The newly launched application will be launched in parallel with the running workflow.

• Use case 1:

• Engineers who launch very long processes on HPC for complex simulations could visualize in real time the output during the execution of the process. The diagram below shows a bioscience simulation using Gromacs.

• Users can access Gromacs in real time to visualize the simulation as it happens.

• Use case 2:

• Data Scientists or AI Engineers launching long ML model training or ML model optimization could visualize the early output in real-time during the execution process using for example TensorBoard. They could then decide to terminate the process or alternatively use the new “Action/signal” feature to change the parameters for the model input.

Feature 3:

• Data Spaces advanced search and filtering:
  ActiveEon’s Data Space is a secure and powerful way to share information between teams during the different stages of a pipeline. Because of the success of this feature, we further improved it by adding powerful search and filtering capabilities.
• This new function corresponds to the trend by our customers to use the platform to automate more sophisticated workloads, specifically for hybrid and AI pipelines, and requires many teams to collaborate.

Feature 4:

• Advanced improved user interface for job monitoring:
  Information tab - Users can see information about jobs execution status as well as the job variable (input parameters) used when the workflow has been launched.
  

Example of parameters for provisioning Spark services through Azure HDInsight

The latter becomes even more important since administrators can allow specific workflow parameters to be changed dynamically.
As described in feature 2, users could quickly glance at the original input parameters and change them accordingly if needed.

Feature 5:

• Workflow tab:

Real-time visualization of workflow execution.
Workflow visualization helps improve business operations. In fact, there is no better way to troubleshoot and improve than by identifying where the process went wrong.
As ActiveEon simplifies the process of measuring progress and success, it becomes easier for teams to spot issues and opportunities for improvement.

ActiveEon displays the workflow which is being executed and graphically shows the progress – the key benefit for operations is being able to pinpoint immediately where the job has failed or paused.

Feature 6:

• Pipeline monitoring.
• Pipelines are at the top level of the workflow hierarchy. An IT procurement pipeline might contain several workflows (provisioning a VM, allocating an IP address…) that in turn include sequential tasks (connecting to the server, creating VM…).
• ActiveEon’s customers have been using pipeline management for many years, however, with the growth of advanced automation like hybrid and AI management, we decided to initiate a new advanced monitoring project and, the new “pipeline monitoring” project represents the first feature of this ambitious strategic plan.
• We are devoting serious efforts to offer our customers a growing number of features that will enable complete end-to-end visibility of pipelines – wherever the process runs and whatever applications are used.
  

In the above screenshot, the pipeline 168 – “Azur_HDInsight_Create_Spark_Cluster…” is composed of three workflows 184, 177 and 175. Administrators can monitor the overall pipeline as well as each individual workflow.

• Advanced Role-Based-Access-Control (RBAC) security functions

• Administrators can further secure the ProActive solution with significant new security features.

• The new advanced RBAC feature enables:

• Administrators can limit the access of folders to specific groups or specific individuals.

• Administrators can set up multiple groups of users and set access right to their group folder.

• Administrators can restrict the use of a “pool” of resources to specific users or groups of users.

• Additional pre-built connectors for major players

• Without any coding, administrators can create sophisticated automation with the following applications:

• ServiceNow (ticketing and business workflows),

• Informatica,

• PeopleSoft,

• VMware,

• Azure HDInsight,

• Apache HBase,

• Apache Phoenix,

• FTPS,

• ActiveEon is continually developing more connectors. Do not hesitate to contact us for the latest list.

• Improved Workflow input parameters (Variables) management

• A workflow parameter is a value that you define before the workflow runs. It is used to set values for tasks in the workflow or to set some user-defined mapping parameters. They also reduce the overhead of creating multiple workflows when you need to change certain attributes of a workflow.
  This new management function will help customers who are developing sophisticated workloads that require many parameters. This is often the case for Big Data, Hybrid and AI pipelines.
  This new capability will help administrators to better structure the parameters for quicker and more efficient management.

• Administrators can now group parameters per topics

• Administrators can tag them

• Administrators can also hide them if they are important for the background execution but not important for the user. This allows to provide a cleaner user interface when users launch their workflows

• This new function corresponds to the trend by our customers to use the platform to automate more sophisticated workloads, specifically for hybrid and AI pipelines and allowing “citizen users” accessing the platform to launch their own workflows

ProActive AI Orchestration

• ProActive AI new “feature engineering” Module

• Data Scientists can now use the “AutoFeat” module during the Feature Engineering process. Feature Engineering is a very important step in machine learning, and it refers to the process of extracting relevant features from the data to train ML algorithms.

• One of the key features of the ProActive AutoFeat is how the algorithm can semi-automatically identify the best method for encoding each categorical column of data, with validation from a Data Scientist during the process.
  Alternatively, Data Scientists can select the data encoding methods they prefer: Label, OneHot, Dummy, Binary, Base N, Hash, and Target.

AutoFeat is used right at the beginning of the model training – AutoFeat will help Data Scientists in this experimental phase by allowing them to select a relevant encoding method and by creating data transformation.

• Once the data frame is finalized, it can be sent to either a feature store solution such as: Feast, Hopsworks or used directly to train an ML model.

• ProActive – “Model as a Service” can be used over multiple models

• AI engineers can now deploy multiple models at the same time, as well as deploy the same model with different versions.

• This version will enable enterprises to manage registry and serving for multiple models – and or multiple versions of the same model.

• ProActive Dashboard for iterative model deployment has been improved for model comparisons

• Once the model is deployed – and needs to be reviewed due, for example, to drifting - an AI engineer is now able to compare the baseline data (metadata) and analyze if the new model is better or worse than the previous deployed one.

• The dashboard is able to display diagrams (plots) comparing the several versions of the same model.

• More powerful use of JupyterLab

• As seen above, a Data scientist can easily launch an Instance of JupyterLab directly from ActiveEon. Our latest development provides even more power to Data Scientists by using pragmas (small directives specified on the notebook cells) directly with the notebook code.
  In other words, Data Scientists running JupyterLab on their laptops can take advantage of the entire available enterprise hybrid compute resources by simply using pragmas and be able to train their models in very powerful machines without lifting a finger.
  Example of JupyterLab coding with #% pragma integrated within Jupyter Notebook:

• We have developed a list of extensive pragma types so that Data Scientists can take control of the way their models are launched, data visualized, process stopped and so on. The key element is that Data Scientists can access the entire enterprise compute resources or, at a push of a button, provision new ones. This allows to abstract the whole complexity and eases the automatic resource provisioning.

• New Dashboard for model monitoring

• Along with the extensive set of features, we have already developed around model deployment, model serving, and model drift, we have just launched our first ModelOps dashboard.
  

The dashboards enable to manage audit and traceability, Analysis of dataset, Prediction review, as well as drift analysis.

It is the first step towards answering customers who are starting to deploy 10s or 100s of models with our framework - and it will be followed very soon by some nifty ModelOps reporting tools – stay tuned.

---