Document how the NGINX Ingress controller build nginx.conf #2479
Conversation
k8s-ci-robot
added
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: aledbf The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
approved
docs/how-it-works.md
Outdated
|
|
||
| ## NGINX configuration | ||
|
|
||
| The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). This is something we need to do, at least until [this PR is merged in OpenResty](0). The main implication of this requirement is the need to reload NGINX after a change in the configuration file. |
There was a problem hiding this comment.
We can probably exclude:
This is something we need to do, at least until [this PR is merged in OpenResty](0). The main implication of this requirement is the need to reload NGINX after a change in the configuration file.
since, the Ingress Controller is always going to assemble a config file, in any case. We can add that PR to another section detailing what it does.
docs/how-it-works.md
Outdated
| ## NGINX model | ||
|
|
||
| Usually, a Kubernetes Controllers utilizes the [synchronization loop pattern](1) to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. | ||
| To get this object from the cluster we use [Kubernetes Informers](2), in particular `FilteredSharedInformer`. This informers allows reacting to changes in using [callbacks](3) to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect or not the final configuration file. This is one of the uses of the model is avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. |
docs/how-it-works.md
Outdated
| Usually, a Kubernetes Controllers utilizes the [synchronization loop pattern](1) to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. | ||
| To get this object from the cluster we use [Kubernetes Informers](2), in particular `FilteredSharedInformer`. This informers allows reacting to changes in using [callbacks](3) to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect or not the final configuration file. This is one of the uses of the model is avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. | ||
|
|
||
| The final representation of the model is generated from a Go template and the model that reflect the NGINX configuration. |
There was a problem hiding this comment.
We can provide a link to the template. We can add some information about template functions as well.
There was a problem hiding this comment.
Also it would be good to add how annotations and cm changes alter the template.
|
|
||
| Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a [work queue](4) it is possible to not lose changes and remove the use of [sync.Mutex](5) to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the [work queue](4). | ||
|
|
||
| Operations to build the model: |
| Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a [work queue](4) it is possible to not lose changes and remove the use of [sync.Mutex](5) to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the [work queue](4). | ||
|
|
||
| Operations to build the model: | ||
|
|
There was a problem hiding this comment.
If the same annotations already existed for a particular path, old ones win.
docs/how-it-works.md
Outdated
|
|
||
| The next list describe the scenarios when a reload is required: | ||
|
|
||
| - New Ingress rule. |
aledbf
removed
the
do-not-merge/work-in-progress
aledbf
changed the title
Codecov Report
@@ Coverage Diff @@
## master #2479 +/- ##
=======================================
Coverage 40.63% 40.63%
=======================================
Files 74 74
Lines 5094 5094
=======================================
Hits 2070 2070
Misses 2743 2743
Partials 281 281Continue to review full report at Codecov.
|
docs/how-it-works.md
Outdated
|
|
||
| ## NGINX model | ||
|
|
||
| Usually, a Kubernetes Controllers utilizes the [synchronization loop pattern](1) to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. |
There was a problem hiding this comment.
nitpick: extra s in "Kubernetes Controllers"
There was a problem hiding this comment.
in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster
this is the complete list of objects ingress-nginx watches, no? What about
To this purpose, we need to build a model using different objects from the cluster. ingress-nginx controller in particular watches (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster.
docs/how-it-works.md
Outdated
| ## NGINX model | ||
|
|
||
| Usually, a Kubernetes Controllers utilizes the [synchronization loop pattern](1) to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. | ||
| To get this object from the cluster we use [Kubernetes Informers](2), in particular `FilteredSharedInformer`. This informers allows reacting to changes in using [callbacks](3) to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. This is one of the uses of the model is avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. |
There was a problem hiding this comment.
reacting to changes in using callbacks to individual changes when a new
to changes in using seems to be redundant
There was a problem hiding this comment.
after
Unfortunately, there is no way to know if a particular change is going to affect the final configuration file.
maybe make it explicit that we have to rebuild a new model from scratch? something like
Therefore on every change we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one then we avoid generating a new Nginx configuration and reloading workers. Otherwise we generate a new Nginx configuration based on the new model, replace the current model and reload Nginx workers.
docs/how-it-works.md
Outdated
| ## NGINX model | ||
|
|
||
| Usually, a Kubernetes Controller utilizes the [synchronization loop pattern](1) to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster. This Ingress controller in particular watches (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. | ||
| To get this object from the cluster we use [Kubernetes Informers](2), in particular `FilteredSharedInformer`. This informers allow reacting [callbacks](3) to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. This is one of the uses of the model is avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. |
There was a problem hiding this comment.
Please check my other comment at #2479 (comment). It was lost after your commit.
docs/how-it-works.md
Outdated
| Usually, a Kubernetes Controller utilizes the [synchronization loop pattern](1) to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster. This Ingress controller in particular watches (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. | ||
| To get this object from the cluster we use [Kubernetes Informers](2), in particular `FilteredSharedInformer`. This informers allow reacting [callbacks](3) to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. This is one of the uses of the model is avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. | ||
|
|
||
| The final representation of the model is generated from a [Go template](6) and the model that reflect the NGINX configuration. |
There was a problem hiding this comment.
The final representation of the model
representation of model or the final Nginx configuration? I presumed by model you refer to the data structure that holds all the objects to render the Nginx config template.
--
s/reflect/reflects
docs/how-it-works.md
Outdated
|
|
||
| Operations to build the model: | ||
|
|
||
| - List Ingress rules using the field ResourceVersion, i.e. old rules first. |
There was a problem hiding this comment.
Did you mean "Order Ingress rules by ResourceVersion field"?
docs/how-it-works.md
Outdated
| - If more than one Ingress defines a TLS section for the same host, the oldest rule wins. | ||
| - If multiple Ingresses define an annotation that affects configuration of the Server block, the oldest rule wins. | ||
|
|
||
| - Create a list of NGINX Servers (hostname) |
There was a problem hiding this comment.
hostname
"per hostname" would be clearer IMO
docs/how-it-works.md
Outdated
|
|
||
| ## When a reload is required | ||
|
|
||
| The next list describe the scenarios when a reload is required: |
docs/how-it-works.md
Outdated
|
|
||
| ## Avoiding reloads | ||
|
|
||
| In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e. a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes. |
There was a problem hiding this comment.
should we reference to enable-dynamic-confioguration docs here?
There was a problem hiding this comment.
Not sure, maybe when we enable enable-dynamic-configuration as the default?
There was a problem hiding this comment.
OK,
In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e. a pod is started or replaced
is a bit confusing then. It sounds like we are saying ingress-nginx already avoids reload in those cases by default - but it does not.
docs/how-it-works.md
Outdated
|
|
||
| One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. | ||
|
|
||
| The final representation of the model is generated from a [Go template](6) and the model that reflects the NGINX configuration. |
There was a problem hiding this comment.
should not this be final representation of Nginx configuration? I commented at #2479 (comment)
Which issue this PR fixes:
fixes #1943