Can't run integration tests in CircleCI

[brikis98]

Running integration tests in CircleCI or TravisCI will fail. This is because the integration tests need to run Boot2Docker in a VM, but in a CI environment, the tests are already running in a VM, and nested VMs are not supported in most environments. See comments below for more details.

[brikis98]

After more research, here is the explanation:

1. Boot2Docker is a 64 bit image with no plans to create a 32 bit image.
2. You can only run 64 bit images if VT-X is enabled.
3. CircleCI and TravisCI both run your code in a VM, so when you try to test docker-osx-dev in the CI environment, you're dealing with a VM-in-a-VM (nested VMs), and you need the "outer" VM to support VT-X.
4. Some hypervisors can virtualize VT-X, but many, such as VirtualBox, cannot.

Therefore, there may not be any way to test this script in a virtualized CI environment.

[JKrag]

I am not sure I understand 100% what you are trying to do in your test. Is it an inherent need to run in a VM, i.e. to test that the install works in a clean environment, or are you just hitting a roadblock because the CI servers available use VM's?

In the last case, I might suggest using a Jenkins server for the integration testing. Jenkins by default doesn't run things in virtual machines (unless you actively run the Jenkins slave on a VM).

Can you explain shortly which tests you would want to run, and what output you would be expecting, then I might try to setup a test test :-).

[brikis98]

@JKrag: Heh, I guess I didn't make it very clear what I wanted to do in this bug, thanks for pointing that out :)

My goal is to create an integration test for docker-osx-dev that performs the following steps:

1. Run the setup.sh script to install all the dependencies on a fresh OS X box.
2. Run the docker-osx-dev script to start syncing a test-project folder to the Boot2Docker VM.
3. Run a docker container, such as docker-alpine, and use the -v flag to mount a test-project folder in it.
4. Check that test-project shows up correctly in the docker container.
5. Check that changes to test-project on OS X are propagated correctly into the docker container.

I roughly captured these ideas in the test.sh script. There are many other integration tests that could be added too, but I wanted to start with something straightforward that would cover a lot of the basic use cases.

While I can run this test locally, I'd like it to run as part of a CI job after every commit and PR. Unfortunately, most free CI hosts do not offer OS X as an operating system, and those that do--namely, Circle CI and Travis CI--run your tests in a VM, which makes it impossible to fire up the Boot2Docker VM within it. I could create my own server environment with Jenkins, but docker-osx-dev is just a side project, so I'd prefer to stick with free, supported CI environments.

If you have any ideas, let me know. Otherwise, it may be better to focus on creating unit tests instead of integration tests, as described in #27.

[Simperfit]

Is this circleCI VM a unix one ?
Can't we just run a docker image of OSX (it may be long to download) ? then run all the things in it :p?

[ain]

@brikis98 is this ticket up to date? What's the latest status?

[brikis98]

@Simperfit: it's a VM image of OS X. Running a VM-in-a-VM is the issue.

@ain: The status is unchanged. I do not have a good way to run integration tests in a CI environment at the moment due to the VM-in-a-VM issue.

[brikis98]

See #97 for the latest work to resolve this ticket. Thank you to @ain for pushing this forward. I'll close this ticket for now so all discussion can happen in one place.