"duplicate listen options" config error when using wildcard ingress resource host

[vdavidoff]

If I configure an ingress resource like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: nginx-non-ssl
  namespace: default
spec:
  rules:
  - host: '*.foo.com'
    http:
      paths:
      - backend:
          serviceName: kube-state-metrics
          servicePort: 8080
        path: /

nginx.conf is written successfully, but then I can see from the ingress controller logs that subsequent attempts to generate the config result in the error:

2017/03/01 22:42:51 [emerg] 697#697: duplicate listen options for [::]:80 in /tmp/nginx-cfg996042368:188
nginx: [emerg] duplicate listen options for [::]:80 in /tmp/nginx-cfg996042368:188
nginx: configuration file /tmp/nginx-cfg996042368 test failed

This only seems to happen if the "host" in the ingress resource is a wildcard name, though maybe it happens other times, too, that I'm not running into.

If I make further edits to the ingress resource, they show up, so whatever this is, it doesn't appear to make further updates block. But for some reason between the first successful write and the next edit, config generation fails in this way.

I'm attaching three files:

• Generated working nginx.conf with one non-wilcard host: nginx.conf.non-wildcard-host.OK.txt
• Generated working nginx.conf with one wildcard host: nginx.conf.wildcard-host.OK.txt
• Generated (tmp) nginx.conf with one wildcard host, after initial successful write, with errors: nginx.conf.wildcard-host.BROKEN.txt

Apologies if what's happening here is obvious and I'm just not seeing it, or if what I'm trying to do here isn't meant to be supported.

[gianrubio]

@vdavidoff what is the version of your ingress? I guess this was already solved on #141

[vdavidoff]

NGINX 0.9.0-beta.2 (note that I haven't had a chance to review the issue you pointed me at, yet)

[gianrubio]

I can't reproduce this, did you customize your template? Please paste your ingress config and the ingress deployment.

[vdavidoff]

I have not modified the template.

The only customizations I have done are via the nginx.conf ConfigMap and annotations on the Ingress.

Here's the live Deployment. Note that I built the image from the 0.9.0-beta.2 release tarball, it was not pulled from an existing 3rd party repo or built from Git.

kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  creationTimestamp: 2017-03-01T17:07:28Z
  generation: 3
  labels:
    k8s-app: kubernetes-nginx-ingress
  name: kubernetes-nginx-ingress
  namespace: default
  resourceVersion: "7472354"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/kubernetes-nginx-ingress
  uid: 8c9e76af-fea1-11e6-bc59-00163e215c34
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kubernetes-nginx-ingress
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kubernetes-nginx-ingress
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=default/kubernetes-nginx-ingress
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: REDACTED/nginx-ingress-controller:0.9.0-beta.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: kubernetes-nginx-ingress
        ports:
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          protocol: TCP
        - containerPort: 18080
          hostPort: 18080
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 60
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2017-03-01T17:07:28Z
    lastUpdateTime: 2017-03-01T17:07:28Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  replicas: 1
  updatedReplicas: 1

Here's the live ConfigMap

apiVersion: v1
data:
  enable-vts-status: "true"
kind: ConfigMap
metadata:
  creationTimestamp: 2017-03-01T17:07:11Z
  name: kubernetes-nginx-ingress
  namespace: default
  resourceVersion: "7472253"
  selfLink: /api/v1/namespaces/default/configmaps/kubernetes-nginx-ingress
  uid: 825de9ad-fea1-11e6-80cf-00163e215c2d

Here's the live Ingress - this is the version that results in the errors I'm reporting. Though again, to be clear, the config was written successfully once (what is described in this Ingress is reflected in the live nginx.conf), and if I modify the Ingress, those changes do appear. Only after the config is initially written do the errors start to appear.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  creationTimestamp: 2017-02-07T17:50:23Z
  generation: 26
  name: nginx-non-ssl
  namespace: default
  resourceVersion: "7474656"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/nginx-non-ssl
  uid: e66e27b2-ed5d-11e6-a32d-00163e215c33
spec:
  rules:
  - host: '*.foo.com'
    http:
      paths:
      - backend:
          serviceName: kube-state-metrics
          servicePort: 8080
        path: /
status:
  loadBalancer:
    ingress:
    - ip: 10.0.64.218

I see the changes from issue #141 in the template in the running container:

root@kubernetes-nginx-ingress-1683316109-fbqsm:/# grep backlogSize /etc/nginx/template/nginx.tmpl                                                                                                                                                 
    {{ $backlogSize := .BacklogSize }}
        listen [::]:80{{ if $cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $index 0 }} ipv6only=off{{end}}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}};
        {{ if not (empty $server.SSLCertificate) }}listen 442 {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}} ssl {{ if $cfg.UseHTTP2 }}http2{{ end }};

I also see them applied, as you'd suspect, in the newest config dropping in /tmp that fails validation:

root@kubernetes-nginx-ingress-1683316109-fbqsm:/# egrep 'backlog|listen \[::\]:80' /tmp/nginx-cfg602239601 
        listen [::]:80 ipv6only=off;
        listen [::]:80 default_server reuseport backlog=511;
        listen 442  default_server reuseport backlog=511 ssl http2;
        listen [::]:18080 ipv6only=off default_server reuseport backlog=511;

I am not familiar with reuseport, but if it's only necessary after the initial listen directive, then it appears as though the "listen [::]:80" directives are in the proper order. If order applies here. I do see the note about it only working with kernels 3.9+, but that looks OK:

root@kubernetes-nginx-ingress-1683316109-fbqsm:/# uname -r
3.13.0-44-generic

[gianrubio]

@vdavidoff thanks for the info, I'll take a look on this during the weekend

[aledbf]

I am not familiar with reuseport,

This flag only can be defined once per listen port (that is why is defined in default_server).
The order is not important.

[gianrubio]

@vdavidoff I can confirm the issue on release 0.9.0-beta.2, but I can't reproduce on the master branch, so looks like it's fixed.

You can wait for the next release or build a custom ingress

[vdavidoff]

OK, thanks. I appreciate you digging into this for me.

[aledbf]

@vdavidoff I was able to reproduce this error. The issue is that the nginx template uses the position in the iteration to add a listen option (ipv6only) and we can specify options only in one listen per port. When the server name start with wildcard or numbers the server contains the ipv6only option and then in the default server the other options. This triggers the error.

[vdavidoff]

Great, thanks again.