Use flexible pipeline in Antrea Agent

[wenyingd]

Describe what you are trying to solve
The motivation for a flexible pipeline includes: 1) each new feature can define its expected pipeline_template, and not worry the conflict with flow installed by other feature; 2) the final pipeline realized on OVS depends on the working features which Antrea Agen is using, and unused flows are not installed at all; 3) new feature no needs to find a valid new table id for its functions; 4) it provides a capability to run Antrea Agent as a new working role (like VM/BM agent) if we want to leverage the existing code

Describe the solution you have in mind
Following are the principles for the flexible pipeline:

1. Antrea Agent loads the required the features in the runtime according to the configuration, and each feature has defined its pipeline_template, which has described the feature’s expected packet’s walking path (including stages and tables in the stage) on OVS. The final pipeline for a packet is automatically generated according to the set of the chosen pipeline_templates.
2. There are 10 predefined static stages(refer to “Flexible pipeline stages” ), and the order of the stages are fixed. Packets should walk through all the stages in order by default. For some special cases, packet is allowed to skip one or more later stages. Resubmit a packet to a previous stage is forbiddenpro
3. There might be 0 or 1 or many tables in one stage, which depends on the working features when Antrea Agent is running. Packet goes to the next table by default. Packet is allowed to resubmit to tableB from tableA with these precondition: 1) tableA and tableB are defined by the same feature, and 2) tableA and tableB are in the same stage.
4. A table is defined in a pipeline_template by a feature, and can be shared by all other features. A feature declares a table (either a new table or an existing tabled defined by the other feature) in its pipeline_template when it needs to add new OpenFlow entries in that table.
5. Tables declared in the same stage in different pipeline_templates are in the same stage in the final pipeline, and the order of these tables is decided by the order that Antrea Agent loads features. The tables defined in the same pipeline_template in a stage are organized in the same order as the feature declares them, but the order for tables from two different pipeline_templates depends on the order in which Antrea Agent loads the related features.
6. The table name is constant, which is defined by a feature. However, table id is dynamically assigned by Antrea Agent in the runtime. The table ids are consecutive, e.g., 0, 1, 2, 3, …. Since the required features are loaded when Antrea Agent is running, all the tables defined in the pipeline_templates can satisfy the function requirement, and we don’t need to think about the compatibility for future features.
7. OVS register, ct_mark and ct_label are shared. We already split register/ct_label/ct_mark to fields according to the bit range. Features can leverage the fields to indicate the characteristics of the packet, e.g., the Connectivity feature forwards packets to the input port if the HairpinMark is set, and Service feature sets the HairpinMark if the client and the final Server (after DNAT) of traffic are the same.

Describe how your solution impacts user flows
When developing a new feature, define a pipeline_template is required if it needs to program OpenFlows on OVS.

1. Declare the stages in which the feature wants to operate packets.
2. Declare tables in the involved stages.
3. Define new fields or change the existing fields if necessary
4. Program OpenFlow entries using the declared tables and fields.

Describe the main design/architecture of your solution
The flexisible pipeline generates table dynamically. There are 10 static stages in a generic pipeline_template, but table ids are generated dynamically according to the featuers on Agent side. By default, a packet goes through all the stages one by one. Following are the static stages in the generic pipeline_template:

1. Classifier Stage
2. Validation Stage
3. PreRouting Stage
4. EgressSecurity Stage
5. Routing Stage
6. PostRouting Stage
7. Switching Stage
8. IngressSecurity Stage
9. Conntrack Stage
10. Output Stage

Alternative solutions that you considered
In current implementation, the pipeline is fixed, and all the table ids are assigned in advance. It is a bit difficult to extend the pipeline when introducing a new feature or changing Antrea Agent working mode (e.g. working on a VM as Agent), for the developer needs to think about how to be compatible with the existing tables, and how to assign a new id for the new function tables. In some cases, we might also need to change the existing table ids to ensure the new function table could work as expected.
With a flexible pipeline design, dev could pay more attention on the feature itself, and the new functions is taken on what stage. The pipeline itself could ensure the packet goes into his/her function table as long as not dropped by other feature functions.

Design doc: https://docs.google.com/document/d/1dSs2-wMFnSEBvi5JhcdTOA94HGll8H5Dbqb-uSUq77U/edit?usp=sharing

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days