Acceptance CI tests

[njam]

Would be great is to have some acceptance tests, because a lot of the problems are dependent on the host or guest OS' particularities. Of course those can't run on travis.

Is there another free CI service that allows to run Virtualbox VMs?
Otherwise we could look into running a jenkins on some dedicated hosting.

[fh]

@njam This is rather complicated - the basic reason here is that most (at least all I am aware of) run their tests in a container of some sort, and you cant start a vagrant virtualization within an already virtualized env). At work, we currently develop some sort of "CI" for vagrant running on a normal root server - while this could technically work, it obviously wont be free.

As a first step, we maybe could still shell out those tests to be able to work in the respective dev env, and then later test ideas how to run them in a CI env?

[fh]

Just ran into https://github.com/mitchellh/vagrant-spec - maybe thats worth looking into?

[njam]

Thanks for the link! I started to draft two different options:

cucumber: #182

• + Good docu, community, support
• + Can run scenarios with different environments (providers, boxes) - "Scenario Outline"
• - Test isolation needs to be done manually after each test (vagrant destroy)

vagrant-spec: #145

• + Explicitly for vagrant, robust test isolation
• - Outdated and only little docu or examples
• - Can't find good way to run test in different environments

I would rather go with cucumber. What do you think? Or we should go a completely different way?

[fh]

Wow, using cucumber here is a cool idea. Totally up for giving it a shot.

[fh]

@njam So, finally was able to resolve my vagrant issues, and gave both approaches a shot. However, the cucumber version still seems not to be fully working, some test cases fail (details in the pr).

I agree with your assessment on vagrant-spec, though, it seems to be not that well suited for our cases here. I would suggest to close the vagrant-spec pr, and focus on cucumber, to get it in a better/fully running state. Not sure how to get this running within travis since a vm cant run in a vm, but maybe we can come up with another solution for a CI env.

What do you think, how should we proceed from here?

[hferentschik]

Merged pull request #182.