Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ingress controller crash on boot - arm arch #2547

Closed
sazo opened this issue May 20, 2018 · 2 comments · Fixed by #2588
Closed

Ingress controller crash on boot - arm arch #2547

sazo opened this issue May 20, 2018 · 2 comments · Fixed by #2588

Comments

@sazo
Copy link

sazo commented May 20, 2018 •
edited
Loading

Is this a BUG REPORT :

NGINX Ingress controller version: kubernetes-ingress-controller/nginx-ingress-controller-arm:0.13.0

Kubernetes version (use kubectl version): 1.9.7

Environment:

  • Cloud provider or hardware configuration: Raspberry PI 3
  • OS: Raspbian GNU/Linux 8 (jessie)
  • Kernel : Linux node01 4.4.50-hypriotos-v7+ Basic structure  #1 SMP PREEMPT Sun Mar 19 14:11:54 UTC 2017 armv7l GNU/Linux
  • Install tools:
  • Others:

What happened:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.13.0
  Build:      git-4bc943a
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

W0520 20:55:29.923638       7 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0520 20:55:29.925445       7 main.go:181] Creating API client for https://10.96.0.1:443
I0520 20:55:30.239565       7 main.go:225] Running in Kubernetes Cluster version v1.9 (v1.9.7) - git (clean) commit dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92 - platform linux/arm
I0520 20:55:30.253145       7 main.go:84] validated ingress-nginx/default-http-backend as the default backend
I0520 20:55:36.221536       7 stat_collector.go:77] starting new nginx stats collector for Ingress controller running in namespace  (class nginx)
I0520 20:55:36.221655       7 stat_collector.go:78] collector extracting information from port 18080
I0520 20:55:36.346751       7 nginx.go:281] starting Ingress controller
I0520 20:55:36.412045       7 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"0c155dfd-eb47-11e7-98ff-b827eba8746c", APIVersion:"v1", ResourceVersion:"2049176", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I0520 20:55:36.425685       7 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"22fe0f2f-eb47-11e7-98ff-b827eba8746c", APIVersion:"v1", ResourceVersion:"2585298", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0520 20:55:36.426531       7 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"a567c4bb-eb47-11e7-98ff-b827eba8746c", APIVersion:"v1", ResourceVersion:"2585299", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0520 20:55:37.474036       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"transmission-ingress", UID:"884303b9-f340-11e7-ae1a-b827eba8746c", APIVersion:"extensions", ResourceVersion:"20547944", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/transmission-ingress
I0520 20:55:37.476615       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"hypriot", UID:"26e3327e-5c45-11e8-a337-b827eba8746c", APIVersion:"extensions", ResourceVersion:"20547945", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/hypriot
I0520 20:55:37.547960       7 nginx.go:302] starting NGINX process...
I0520 20:55:37.548184       7 leaderelection.go:175] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
W0520 20:55:37.557541       7 controller.go:283] error getting service default/rtorrent: service default/rtorrent was not found
W0520 20:55:37.557734       7 controller.go:283] error getting service default/rtorrent: service default/rtorrent was not found
I0520 20:55:37.557813       7 controller.go:170] backend reload required
I0520 20:55:37.557880       7 stat_collector.go:34] changing prometheus collector from  to default
I0520 20:55:37.566887       7 status.go:196] new leader elected: nginx-ingress-controller-85cfc97b46-vrm8z
E0520 20:55:37.864143       7 controller.go:175] unexpected failure restarting the backend: 

-------------------------------------------------------------------------------
Error: exit status 1
2018/05/20 20:55:37 [error] 34#34: init_by_lua error: /usr/local/lib/lua/resty/waf/log.lua:4: module 'cjson' not found:
	no field package.preload['cjson']
	no file '/etc/nginx/lua/cjson.lua'
	no file '/etc/nginx/lua/vendor/cjson.lua'
	no file '/usr/local/lib/lua/cjson.lua'
	no file './cjson.lua'
	no file '/usr/local/share/luajit-2.1.0-beta3/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson/init.lua'
	no file '/usr/local/lib/lua/cjson.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.1/cjson.so'
	no file './cjson.so'
	no file '/usr/local/lib/lua/5.1/cjson.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/log.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/actions.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf.lua:3: in main chunk
	[C]: in function 'require'
	init_by_lua:5: in main chunk
nginx: [error] init_by_lua error: /usr/local/lib/lua/resty/waf/log.lua:4: module 'cjson' not found:
	no field package.preload['cjson']
	no file '/etc/nginx/lua/cjson.lua'
	no file '/etc/nginx/lua/vendor/cjson.lua'
	no file '/usr/local/lib/lua/cjson.lua'
	no file './cjson.lua'
	no file '/usr/local/share/luajit-2.1.0-beta3/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson/init.lua'
	no file '/usr/local/lib/lua/cjson.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.1/cjson.so'
	no file './cjson.so'
	no file '/usr/local/lib/lua/5.1/cjson.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/log.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/actions.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf.lua:3: in main chunk
	[C]: in function 'require'
	init_by_lua:5: in main chunk
nginx: configuration file /tmp/nginx-cfg631025675 test failed

-------------------------------------------------------------------------------
W0520 20:55:37.865086       7 queue.go:113] requeuing , err 
-------------------------------------------------------------------------------
Error: exit status 1

-------- ALOT MORE ---------

nginx: [error] init_by_lua error: /usr/local/lib/lua/resty/waf/log.lua:4: module 'cjson' not found:
	no field package.preload['cjson']
	no file '/etc/nginx/lua/cjson.lua'
	no file '/etc/nginx/lua/vendor/cjson.lua'
	no file '/usr/local/lib/lua/cjson.lua'
	no file './cjson.lua'
	no file '/usr/local/share/luajit-2.1.0-beta3/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson/init.lua'
	no file '/usr/local/lib/lua/cjson.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.1/cjson.so'
	no file './cjson.so'
	no file '/usr/local/lib/lua/5.1/cjson.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/log.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/actions.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf.lua:3: in main chunk
	[C]: in function 'require'
	init_by_lua:5: in main chunk
nginx: configuration file /tmp/nginx-cfg316304495 test failed

-------------------------------------------------------------------------------
W0520 20:55:54.224484       7 controller.go:283] error getting service default/rtorrent: service default/rtorrent was not found
W0520 20:55:54.224659       7 controller.go:283] error getting service default/rtorrent: service default/rtorrent was not found
I0520 20:55:54.224984       7 controller.go:170] backend reload required
E0520 20:55:54.519124       7 controller.go:175] unexpected failure restarting the backend: 

-------------------------------------------------------------------------------
Error: exit status 1
2018/05/20 20:55:54 [error] 70#70: init_by_lua error: /usr/local/lib/lua/resty/waf/log.lua:4: module 'cjson' not found:
	no field package.preload['cjson']
	no file '/etc/nginx/lua/cjson.lua'
	no file '/etc/nginx/lua/vendor/cjson.lua'
	no file '/usr/local/lib/lua/cjson.lua'
	no file './cjson.lua'
	no file '/usr/local/share/luajit-2.1.0-beta3/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson/init.lua'
	no file '/usr/local/lib/lua/cjson.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.1/cjson.so'
	no file './cjson.so'
	no file '/usr/local/lib/lua/5.1/cjson.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/log.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/actions.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf.lua:3: in main chunk
	[C]: in function 'require'
	init_by_lua:5: in main chunk
nginx: [error] init_by_lua error: /usr/local/lib/lua/resty/waf/log.lua:4: module 'cjson' not found:
	no field package.preload['cjson']
	no file '/etc/nginx/lua/cjson.lua'
	no file '/etc/nginx/lua/vendor/cjson.lua'
	no file '/usr/local/lib/lua/cjson.lua'
	no file './cjson.lua'
	no file '/usr/local/share/luajit-2.1.0-beta3/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson/init.lua'
	no file '/usr/local/lib/lua/cjson.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.1/cjson.so'
	no file './cjson.so'
	no file '/usr/local/lib/lua/5.1/cjson.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/log.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/actions.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf.lua:3: in main chunk
	[C]: in function 'require'
	init_by_lua:5: in main chunk
nginx: configuration file /tmp/nginx-cfg395998466 test failed

-------------------------------------------------------------------------------
W0520 20:55:54.519263       7 queue.go:113] requeuing monitoring/prometheus-node-exporter, err 
-------------------------------------------------------------------------------
Error: exit status 1
2018/05/20 20:55:54 [error] 70#70: init_by_lua error: /usr/local/lib/lua/resty/waf/log.lua:4: module 'cjson' not found:
	no field package.preload['cjson']
	no file '/etc/nginx/lua/cjson.lua'
	no file '/etc/nginx/lua/vendor/cjson.lua'
	no file '/usr/local/lib/lua/cjson.lua'
	no file './cjson.lua'
	no file '/usr/local/share/luajit-2.1.0-beta3/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson/init.lua'
	no file '/usr/local/lib/lua/cjson.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.1/cjson.so'
	no file './cjson.so'
	no file '/usr/local/lib/lua/5.1/cjson.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/log.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/actions.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf.lua:3: in main chunk
	[C]: in function 'require'
	init_by_lua:5: in main chunk
nginx: [error] init_by_lua error: /usr/local/lib/lua/resty/waf/log.lua:4: module 'cjson' not found:
	no field package.preload['cjson']
	no file '/etc/nginx/lua/cjson.lua'
	no file '/etc/nginx/lua/vendor/cjson.lua'
	no file '/usr/local/lib/lua/cjson.lua'
	no file './cjson.lua'
	no file '/usr/local/share/luajit-2.1.0-beta3/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson.lua'
	no file '/usr/local/share/lua/5.1/cjson/init.lua'
	no file '/usr/local/lib/lua/cjson.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.1/cjson.so'
	no file './cjson.so'
	no file '/usr/local/lib/lua/5.1/cjson.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/log.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf/actions.lua:4: in main chunk
	[C]: in function 'require'
	/usr/local/lib/lua/resty/waf.lua:3: in main chunk
	[C]: in function 'require'
	init_by_lua:5: in main chunk
nginx: configuration file /tmp/nginx-cfg395998466 test failed

-------------------------------------------------------------------------------
I0520 20:55:57.449633       7 main.go:150] Received SIGTERM, shutting down
I0520 20:55:57.449790       7 nginx.go:365] shutting down controller queues
I0520 20:55:57.449946       7 nginx.go:373] stopping NGINX process...
2018/05/20 20:55:57 [notice] 71#71: signal process started

What you expected to happen:
The ingress controller starts

How to reproduce it (as minimally and precisely as possible):
Boot the ingress controller on arm architecture

Anything else we need to know:
Same error comes when trying 0.15, 0.14. 0.13, 0.12
@toolboc
Copy link
Contributor

toolboc commented May 29, 2018

@sazo , looks like the image is missing lua-cjson as required by lua-resty-waf

This needs to be fixed in the arm build script, for now you can hop into your container and run:

apt update
apt get install luarocks
luarocks install lua-cjson

or pull my image which handles this dependency:

docker pull toolboc/nginx-ingress-controller-arm:0.15.0

@aledbf aledbf mentioned this issue May 31, 2018
Merged
toolboc added a commit to toolboc/ingress-nginx that referenced this issue May 31, 2018
@toolboc
Fix issue kubernetes#2547 - missing lua-cjson dependency
139b8df
@toolboc toolboc mentioned this issue May 31, 2018
Closed
@sazo
Copy link
Author

sazo commented May 31, 2018

@toolboc Thanks! Looking forward to see the PR merged :)

This was referenced Jun 2, 2018
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update nginx dependencies aledbf/ingress-nginx
2 participants
@sazo @toolboc