openshift-ansible integration test failures; Failed to get D-Bus connection: Operation not permitted

[sosiouxme]

openshift-ansible merge/test includes integration tests to exercise the pre-install checks. These are failing quite often but not 100% of the time.

Examples

1 2 3

Current Result

All tests fail with similar problem:

--- FAIL: TestPackageUpdateDepMissing (114.93s)
	common.go:54: missing in output: ["check \"package_update\":" "Could not perform a yum update." "break-yum-update-1.0-2.noarch requires package-that-does-not-exist"]
	common.go:98: 
		$ (cd /data/src/github.com/openshift/openshift-ansible/test/integration/openshift_health_checker/preflight && ansible-playbook -i /dev/null playbooks/package_update_dep_missing.yml)
		...
		localhost                  : ok=15   changed=2    unreachable=0    failed=0   
		openshift_ansible_test_20198306014053 : ok=43   changed=6    unreachable=0    failed=1   
		
		
		Failure summary:
		
		  1. Host:     openshift_ansible_test_20198306014053
		     Play:     Initialize host facts
		     Task:     os_firewall : Ensure firewalld service is not enabled
		     Message:  Failed to get D-Bus connection: Operation not permitted

Additional Information

These tests are somewhat unique in that they run using containers as install target hosts. I suspect that some change around docker has occurred, blocking a path that used to work. I haven't seen the same failure yet when running the tests locally.

[sdodson]

Yeah this is happening on all openshift_ansible test and merge jobs :-(

[sdodson]

Looking through the logs the version of docker that it's using is pre RHEL 7.4 so it's not particularly new.

[kwoodson]

@sdodson, @sosiouxme, The reason this is happening is due to the refactor of the firewall into the std_include.yml. The std_include.yml is happening in the beginning to initialize whether we want iptables or firewalld. Depending on which one is desired the os_firewall role attempts to start the service. This service fails with the error message (locally):

fatal: [openshift_ansible_test_51223548473159]: FAILED! => {
    "changed": false, 
    "cmd": "/usr/bin/systemctl", 
    "failed": true, 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "daemon_reload": false, 
            "enabled": false, 
            "masked": true, 
            "name": "firewalld", 
            "no_block": false, 
            "state": "stopped", 
            "user": false
        }
    }, 
    "msg": "Failed to get D-Bus connection: Operation not permitted", 
    "rc": 1, 
    "stderr": "Failed to get D-Bus connection: Operation not permitted\n", 
    "stderr_lines": [
        "Failed to get D-Bus connection: Operation not permitted"
    ], 
    "stdout": "", 
    "stdout_lines": []
}

This is a behavior change. I believe that we are planning to move that out of the way with this openshift/openshift-ansible#5084. If it is removed out of the std_include.yml then this should work.

The overall issue is that the container does not have permission to control the host's services. It either needs to be a privileged container, overlay with some other options, etc. https://forums.docker.com/t/any-simple-and-safe-way-to-start-services-on-centos7-systemd/5695

[sosiouxme]

OK. Well I don't understand how it's happening most of the time but not all the time, and I can't reproduce locally. Hope this PR helps, I'll keep looking into it.

…

On Tue, Aug 15, 2017 at 8:33 AM, Scott Dodson ***@***.***> wrote: Looking through the logs the version of docker that it's using is pre RHEL 7.4 so it's not particularly new. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#15769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABz-rNxW83oBaSYxdyaNuAXC8lXaXmtks5sYZApgaJpZM4O2vXk> .

[sosiouxme]

openshift/openshift-ansible#5084 is merged and it looks like tests are succeeding.