Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx uses previous configuration #306

Closed
kobrinartem opened this issue Jul 11, 2018 · 13 comments
Closed

Nginx uses previous configuration #306

kobrinartem opened this issue Jul 11, 2018 · 13 comments
Labels
bug An issue reporting a potential bug

Comments

@kobrinartem
Copy link

kobrinartem commented Jul 11, 2018

I am swapping between two version of application.

Master Ingress
`---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: default-nginx-ingress-plus
namespace: blue-green-demo
annotations:
kubernetes.io/ingress.class: "external-plus-ingress"
nginx.org/mergeable-ingress-type: "master"

spec:
rules:

  • host: plus.example.com
    `

App Config
`---

apiVersion: v1
kind: Service
metadata:
name: echoheaders-blue
namespace: blue-green-demo
labels:
app: echoheaders-blue
spec:
type: NodePort
ports:

  • port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  • port: 443
    targetPort: 8443
    protocol: TCP
    name: https
    selector:
    app: echoheaders
    version: blue

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoheaders-blue
namespace: blue-green-demo
labels:
app: echoheaders
version: blue
spec:
replicas: 3
selector:
matchLabels:
app: echoheaders
version: blue
template:
metadata:
labels:
app: echoheaders
version: blue
spec:
containers:
- name: echoheaders
image: gcr.io/google_containers/echoserver:1.10
ports:
- containerPort: 8080
- containerPort: 8443
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: external-echoheaders-sticky-ingress-plus
namespace: blue-green-demo
annotations:
kubernetes.io/ingress.class: "external-plus-ingress"
nginx.org/mergeable-ingress-type: "minion"
nginx.com/sticky-cookie-services: "serviceName=echoheaders-blue srv_id expires=1h path=/echo"

spec:
rules:

  • host: plus.example.com
    http:
    paths:
    • backend:
      serviceName: echoheaders-blue
      servicePort: 80
      path: /echo`

kubectl patch -n blue-green-demo svc echoheaders-blue -p "{"spec":{"selector": {"app": "echoheaders", "version": "green"}}}"

@kobrinartem
Copy link
Author

I see that controller has new config file. But nginx uses previous config file. I fixed it by nginx -s reload

@kobrinartem
Copy link
Author

/nginx-ingress -version
Version=1.2.0 GitCommit=ddad8b5

@Dean-Coakley
Copy link
Contributor

Dean-Coakley commented Jul 12, 2018

@kobrinartem I'm trying to reproduce this issue.

Can you provide me with logs from the Ingress controller?

I don't see the git commit ddad8b5 in our master branch, where have you installed the Ingress controller from?

@kobrinartem
Copy link
Author

Logs right after updating selector tags in service. Nginx has newest version in config but uses previous version

2018/07/13 10:56:31 [notice] 12#12: signal 1 (SIGHUP) received from 73, reconfiguring
2018/07/13 10:56:31 [notice] 12#12: reconfiguring
I0713 10:56:31.746992       1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for blue-green-demo/external-echoheaders-sticky-ingress-plus(Minion) was added or updated
I0713 10:56:31.747041       1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for blue-green-demo/default-nginx-ingress-plus(Master) was added or updated
W0713 10:56:31.747241       1 controller.go:825] Error retrieving endpoints for the service echoheaders-blue: No endpoints for target port 8080 in service echoheaders-blue
I0713 10:56:31.747494       1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for ingress-plus-nginx/external-nginx-state-sticky-ingress-plus(Minion) was added or updated
I0713 10:56:31.748134       1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for blue-green-demo/external-echoheaders-stage-sticky-ingress-plus(Minion) was added or updated
2018/07/13 10:56:31 [notice] 12#12: using the "epoll" event method
2018/07/13 10:56:31 [notice] 12#12: start worker processes
2018/07/13 10:56:31 [notice] 12#12: start worker process 74
2018/07/13 10:56:31 [notice] 12#12: start worker process 75
2018/07/13 10:56:31 [notice] 63#63: gracefully shutting down
2018/07/13 10:56:31 [notice] 62#62: gracefully shutting down
2018/07/13 10:56:31 [notice] 62#62: exiting
2018/07/13 10:56:31 [notice] 63#63: exiting
2018/07/13 10:56:31 [notice] 62#62: exit
2018/07/13 10:56:31 [notice] 63#63: exit
2018/07/13 10:56:31 [notice] 12#12: signal 17 (SIGCHLD) received from 62
2018/07/13 10:56:31 [notice] 12#12: worker process 62 exited with code 0
2018/07/13 10:56:31 [notice] 12#12: signal 29 (SIGIO) received
2018/07/13 10:56:31 [notice] 12#12: signal 17 (SIGCHLD) received from 63
2018/07/13 10:56:31 [notice] 12#12: worker process 63 exited with code 0
2018/07/13 10:56:31 [notice] 12#12: signal 29 (SIGIO) received
W0713 10:58:05.566710       1 reflector.go:341] github.com/nginxinc/kubernetes-ingress/nginx-controller/controller/controller.go:353: watch of *v1.ConfigMap ended with: too old resource version: 12280274 (12280837)
32.42.29.204 - - [13/Jul/2018:10:58:41 +0000] "GET /echo HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" "-"```

@Dean-Coakley
Copy link
Contributor

@kobrinartem Sorry, I should have asked for this previously, but can you please set logging verbosity to 3 and provide me with those logs?

@kobrinartem
Copy link
Author

kobrinartem commented Jul 13, 2018

@Dean-Coakley
32.42.29.204 - - [13/Jul/2018:11:55:19 +0000] "GET /echo HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" "-"
I0713 11:55:20.690087 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:20.690111 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:20.690137 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:20.690145 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:20.690151 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:20.873992 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:20.874012 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:20.874024 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:20.874029 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:20.874034 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:21.195742 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:21.195765 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:21.195777 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:21.195781 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:21.195786 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
I0713 11:55:22.694947 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:22.694975 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:22.694993 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:22.694997 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:22.695002 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:22.974708 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:22.974730 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:22.974747 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:22.974752 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:22.974757 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:23.205100 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:23.205123 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:23.205165 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:23.205194 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:23.205214 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
I0713 11:55:24.708189 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:24.708213 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:24.708230 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:24.708235 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:24.708239 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:24.981866 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:24.981890 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:24.981908 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:24.981913 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:24.981918 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:25.209514 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:25.209538 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:25.209597 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:25.209623 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:25.209633 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
I0713 11:55:26.717038 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:26.717060 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:26.717101 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:26.717110 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:26.717116 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:26.987909 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:26.987932 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:26.987971 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:26.987991 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:26.988014 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:27.215931 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:27.215953 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:27.215966 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:27.215970 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:27.215975 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
I0713 11:55:28.723952 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:28.723977 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:28.723994 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:28.723999 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:28.724006 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:28.991396 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:28.991494 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:28.991527 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:28.991547 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:28.991564 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:29.220804 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:29.220827 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:29.220844 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:29.220850 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:29.220854 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
I0713 11:55:30.729131 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:30.729153 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:30.729190 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:30.729199 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:30.729204 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:31.176105 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:31.176122 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:31.176134 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:31.176138 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:31.176143 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:31.225778 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:31.225798 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:31.225820 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:31.225825 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:31.225830 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
I0713 11:55:32.734626 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:32.734687 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:32.734732 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:32.734754 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:32.734771 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:33.187827 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:33.187912 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:33.187944 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:33.187964 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:33.187981 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:33.230050 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:33.230139 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:33.230178 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:33.230189 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:33.230193 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
I0713 11:55:33.563432 1 controller.go:199] Service echoheaders-blue changed, syncing
I0713 11:55:33.563485 1 utils.go:63] Adding an element with a key: blue-green-demo/default-nginx-ingress-plus
I0713 11:55:33.563498 1 utils.go:89] Syncing blue-green-demo/default-nginx-ingress-plus
I0713 11:55:33.563502 1 controller.go:513] Syncing blue-green-demo/default-nginx-ingress-plus
I0713 11:55:33.563507 1 controller.go:545] Adding or Updating Ingress: blue-green-demo/default-nginx-ingress-plus
I0713 11:55:33.563672 1 nginx.go:200] Updating NGINX configuration

upstream blue-green-demo-external-echoheaders-sticky-ingress-plus-plus.example.com-echoheaders-blue-80 {
zone blue-green-demo-external-echoheaders-sticky-ingress-plus-plus.example.com-echoheaders-blue-80 256k;
least_conn;

server 100.96.10.34:8080 max_fails=1 fail_timeout=10s;
server 100.96.11.47:8080 max_fails=1 fail_timeout=10s;
server 100.96.2.21:8080 max_fails=1 fail_timeout=10s;

sticky cookie srv_id expires=1h path=/echo;

I0713 11:55:33.564482 1 nginx.go:284] Writing NGINX conf to /etc/nginx/conf.d/blue-green-demo-default-nginx-ingress-plus.conf
I0713 11:55:33.565232 1 nginx.go:304] NGINX configuration file had been updated
I0713 11:55:33.565280 1 nginx.go:360] executing nginx -t

}
upstream blue-green-demo-external-echoheaders-stage-sticky-ingress-plus-plus.example.com-echoheaders-green-80 {
zone blue-green-demo-external-echoheaders-stage-sticky-ingress-plus-plus.example.com-echoheaders-green-80 256k;
least_conn;

server 100.96.3.17:8080 max_fails=1 fail_timeout=10s;
server 100.96.6.21:8080 max_fails=1 fail_timeout=10s;
server 100.96.8.23:8080 max_fails=1 fail_timeout=10s;

sticky cookie srv_id expires=1h path=/echo-stage;

}
upstream ingress-plus-nginx-external-nginx-state-sticky-ingress-plus-plus.example.com-external-nginx-state-8080 {
zone ingress-plus-nginx-external-nginx-state-sticky-ingress-plus-plus.example.com-external-nginx-state-8080 256k;
least_conn;

server 100.96.4.35:8080 max_fails=1 fail_timeout=10s;

}

server {

listen 80 proxy_protocol;



set_real_ip_from 0.0.0.0/0;
real_ip_header proxy_protocol;


server_tokens "on";

server_name plus.example.com;

status_zone plus.example.com;











location /echo {
	
	proxy_http_version 1.1;
	

	proxy_connect_timeout 60s;
	proxy_read_timeout 60s;
	client_max_body_size 1m;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Host $host;
	proxy_set_header X-Forwarded-Port $server_port;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_buffering on;
	
	proxy_pass http://blue-green-demo-external-echoheaders-sticky-ingress-plus-plus.example.com-echoheaders-blue-80;
	
	
}
location /echo-stage {
	
	proxy_http_version 1.1;
	

	proxy_connect_timeout 60s;
	proxy_read_timeout 60s;
	client_max_body_size 1m;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Host $host;
	proxy_set_header X-Forwarded-Port $server_port;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_buffering on;
	
	proxy_pass http://blue-green-demo-external-echoheaders-stage-sticky-ingress-plus-plus.example.com-echoheaders-green-80;
	
	
}
location /nginx-state {
	
	proxy_http_version 1.1;
	

	proxy_connect_timeout 60s;
	proxy_read_timeout 60s;
	client_max_body_size 1m;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Host $host;
	proxy_set_header X-Forwarded-Port $server_port;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_buffering on;
	
	proxy_pass http://ingress-plus-nginx-external-nginx-state-sticky-ingress-plus-plus.example.com-external-nginx-state-8080;
	
	
}

}
I0713 11:55:33.573467 1 nginx.go:360] executing nginx -s reload
I0713 11:55:33.575314 1 controller.go:234] Endpoints echoheaders-blue changed, syncing
I0713 11:55:33.575374 1 utils.go:63] Adding an element with a key: blue-green-demo/echoheaders-blue
2018/07/13 11:55:33 [notice] 53#53: signal process started
2018/07/13 11:55:33 [notice] 10#10: signal 1 (SIGHUP) received from 53, reconfiguring
2018/07/13 11:55:33 [notice] 10#10: reconfiguring
2018/07/13 11:55:33 [notice] 10#10: using the "epoll" event method
2018/07/13 11:55:33 [notice] 10#10: start worker processes
2018/07/13 11:55:33 [notice] 10#10: start worker process 54
2018/07/13 11:55:33 [notice] 10#10: start worker process 55
I0713 11:55:33.603624 1 utils.go:89] Syncing blue-green-demo/echoheaders-blue
I0713 11:55:33.603661 1 controller.go:513] Syncing blue-green-demo/echoheaders-blue
I0713 11:55:33.603680 1 controller.go:369] Syncing endpoints blue-green-demo/echoheaders-blue
I0713 11:55:33.605447 1 controller.go:399] Updating Endpoints for blue-green-demo/external-echoheaders-sticky-ingress-plus
I0713 11:55:33.605540 1 nginx.go:200] Updating NGINX configuration
I0713 11:55:33.606236 1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for blue-green-demo/external-echoheaders-sticky-ingress-plus(Minion) was added or updated
I0713 11:55:33.606299 1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for blue-green-demo/default-nginx-ingress-plus(Master) was added or updated
I0713 11:55:33.606334 1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for blue-green-demo/external-echoheaders-stage-sticky-ingress-plus(Minion) was added or updated
I0713 11:55:33.606363 1 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"blue-green-demo", Name:"default-nginx-ingress-plus", UID:"95534a06-8475-11e8-a111-0ab285474f22", APIVersion:"extensions", ResourceVersion:"11580207", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for ingress-plus-nginx/external-nginx-state-sticky-ingress-plus(Minion) was added or updated
I0713 11:55:33.606677 1 nginx.go:284] Writing NGINX conf to /etc/nginx/conf.d/blue-green-demo-default-nginx-ingress-plus.conf

upstream blue-green-demo-external-echoheaders-sticky-ingress-plus-plus.example.com-echoheaders-blue-80 {
zone blue-green-demo-external-echoheaders-sticky-ingress-plus-plus.example.com-echoheaders-blue-80 256k;
least_conn;

server 100.96.3.17:8080 max_fails=1 fail_timeout=10s;
server 100.96.6.21:8080 max_fails=1 fail_timeout=10s;
server 100.96.8.23:8080 max_fails=1 fail_timeout=10s;

sticky cookie srv_id expires=1h path=/echo;

}
upstream blue-green-demo-external-echoheaders-stage-sticky-ingress-plus-plus.example.com-echoheaders-green-80 {
zone blue-green-demo-external-echoheaders-stage-sticky-ingress-plus-plus.example.com-echoheaders-green-80 256k;
least_conn;

server 100.96.3.17:8080 max_fails=1 fail_timeout=10s;
server 100.96.6.21:8080 max_fails=1 fail_timeout=10s;
server 100.96.8.23:8080 max_fails=1 fail_timeout=10s;

sticky cookie srv_id expires=1h path=/echo-stage;

}
upstream ingress-plus-nginx-external-nginx-state-sticky-ingress-plus-plus.example.com-external-nginx-state-8080 {
zone ingress-plus-nginx-external-nginx-state-sticky-ingress-plus-plus.example.com-external-nginx-state-8080 256k;
least_conn;

server 100.96.4.35:8080 max_fails=1 fail_timeout=10s;

}

server {

listen 80 proxy_protocol;



set_real_ip_from 0.0.0.0/0;
real_ip_header proxy_protocol;


server_tokens "on";

server_name plus.example.com;

status_zone plus.example.com;











location /echo {
	
	proxy_http_version 1.1;
	

	proxy_connect_timeout 60s;
	proxy_read_timeout 60s;
	client_max_body_size 1m;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Host $host;
	proxy_set_header X-Forwarded-Port $server_port;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_buffering on;
	
	proxy_pass http://blue-green-demo-external-echoheaders-sticky-ingress-plus-plus.example.com-echoheaders-blue-80;
	
	
}
location /echo-stage {
	
	proxy_http_version 1.1;
	

	proxy_connect_timeout 60s;
	proxy_read_timeout 60s;
	client_max_body_size 1m;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Host $host;
	proxy_set_header X-Forwarded-Port $server_port;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_buffering on;
	
	proxy_pass http://blue-green-demo-external-echoheaders-stage-sticky-ingress-plus-plus.example.com-echoheaders-green-80;
	
	
}
location /nginx-state {
	
	proxy_http_version 1.1;
	

	proxy_connect_timeout 60s;
	proxy_read_timeout 60s;
	client_max_body_size 1m;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Host $host;
	proxy_set_header X-Forwarded-Port $server_port;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_buffering on;
	
	proxy_pass http://ingress-plus-nginx-external-nginx-state-sticky-ingress-plus-plus.example.com-external-nginx-state-8080;
	
	
}

}
I0713 11:55:33.609178 1 nginx.go:304] NGINX configuration file had been updated
I0713 11:55:33.609660 1 nginx_api.go:51] Updated servers of blue-green-demo-external-echoheaders-sticky-ingress-plus-plus.example.com-echoheaders-blue-80; Added: [], Removed: []
I0713 11:55:33.609885 1 nginx_api.go:51] Updated servers of blue-green-demo-external-echoheaders-stage-sticky-ingress-plus-plus.example.com-echoheaders-green-80; Added: [], Removed: []
I0713 11:55:33.610058 1 nginx_api.go:51] Updated servers of ingress-plus-nginx-external-nginx-state-sticky-ingress-plus-plus.example.com-external-nginx-state-8080; Added: [], Removed: []
2018/07/13 11:55:33 [notice] 49#49: gracefully shutting down
2018/07/13 11:55:33 [notice] 48#48: gracefully shutting down
2018/07/13 11:55:33 [notice] 48#48: exiting
2018/07/13 11:55:33 [notice] 49#49: exiting
2018/07/13 11:55:33 [notice] 48#48: exit
2018/07/13 11:55:33 [notice] 49#49: exit
2018/07/13 11:55:33 [notice] 10#10: signal 17 (SIGCHLD) received from 48
2018/07/13 11:55:33 [notice] 10#10: worker process 48 exited with code 0
2018/07/13 11:55:33 [notice] 10#10: worker process 49 exited with code 0
2018/07/13 11:55:33 [notice] 10#10: signal 29 (SIGIO) received
2018/07/13 11:55:33 [notice] 10#10: signal 17 (SIGCHLD) received from 49
I0713 11:55:34.743865 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:34.743889 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:34.743901 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:34.743906 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:34.743911 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:35.195793 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:35.195815 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:35.195830 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:35.195892 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:35.195904 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:35.240141 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:35.240163 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:35.240179 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:35.240212 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:35.240221 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
32.42.29.204 - - [13/Jul/2018:11:55:36 +0000] "GET /echo HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" "-"
I0713 11:55:36.747014 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:36.747035 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:36.747050 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:36.747055 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:36.747060 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:37.210385 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:37.210406 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:37.210475 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:37.210489 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:37.210494 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:37.247658 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:37.247731 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:37.247762 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:37.247809 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:37.247819 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
32.42.29.204 - - [13/Jul/2018:11:55:38 +0000] "GET /echo HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" "-"
I0713 11:55:38.754082 1 controller.go:234] Endpoints kube-scheduler changed, syncing
I0713 11:55:38.754101 1 utils.go:63] Adding an element with a key: kube-system/kube-scheduler
I0713 11:55:38.754119 1 utils.go:89] Syncing kube-system/kube-scheduler
I0713 11:55:38.754124 1 controller.go:513] Syncing kube-system/kube-scheduler
I0713 11:55:38.754129 1 controller.go:369] Syncing endpoints kube-system/kube-scheduler
I0713 11:55:39.217535 1 controller.go:234] Endpoints cluster-autoscaler changed, syncing
I0713 11:55:39.217555 1 utils.go:63] Adding an element with a key: kube-system/cluster-autoscaler
I0713 11:55:39.217570 1 utils.go:89] Syncing kube-system/cluster-autoscaler
I0713 11:55:39.217574 1 controller.go:513] Syncing kube-system/cluster-autoscaler
I0713 11:55:39.217580 1 controller.go:369] Syncing endpoints kube-system/cluster-autoscaler
I0713 11:55:39.253048 1 controller.go:234] Endpoints kube-controller-manager changed, syncing
I0713 11:55:39.253068 1 utils.go:63] Adding an element with a key: kube-system/kube-controller-manager
I0713 11:55:39.253079 1 utils.go:89] Syncing kube-system/kube-controller-manager
I0713 11:55:39.253084 1 controller.go:513] Syncing kube-system/kube-controller-manager
I0713 11:55:39.253142 1 controller.go:369] Syncing endpoints kube-system/kube-controller-manager
32.42.29.204 - - [13/Jul/2018:11:55:40 +0000] "GET /echo HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" "-"

@kobrinartem
Copy link
Author

/nginx-ingress -version
Version=1.2.0 GitCommit=1144d00

@kobrinartem
Copy link
Author

kobrinartem commented Jul 13, 2018

it's command to switch blue to green:
kubectl patch -n blue-green-demo svc echoheaders-blue -p "{\"spec\":{\"selector\": {\"app\": \"echoheaders\", \"version\": \"green\"}}}"

blue pods:

server 100.96.10.34:8080 max_fails=1 fail_timeout=10s;
server 100.96.11.47:8080 max_fails=1 fail_timeout=10s;
server 100.96.2.21:8080 max_fails=1 fail_timeout=10s;

green pods:

server 100.96.3.17:8080 max_fails=1 fail_timeout=10s;
server 100.96.6.21:8080 max_fails=1 fail_timeout=10s;
server 100.96.8.23:8080 max_fails=1 fail_timeout=10s;

@Dean-Coakley
Copy link
Contributor

@kobrinartem I have been able to recreate this issue.

I am working on resolving this for you as soon as possible.

Thanks for reporting this issue and providing all these logs for me. They were very helpful in finding the issue.

@isaachawley isaachawley added the bug An issue reporting a potential bug label Jul 30, 2018
@isaachawley
Copy link
Contributor

We're still working to fix this issue.

@isaachawley
Copy link
Contributor

@kobrinartem quick update: we're still working on this.

@isaachawley
Copy link
Contributor

Hi @kobrinartem,

We've merged in #365 which should fix this issue. Sorry for the delay.

It's not in a release yet, but we do Edge versions now if you'd like to test with a built image.

Thanks

@isaachawley
Copy link
Contributor

Merged #380 which should prevent this type of issue from happening in any case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug An issue reporting a potential bug
Projects
None yet
Development

No branches or pull requests

3 participants