Update the testing matrix (5 distros, 2 python versions, 3 salt versions) #406

javierbertoli · 2019-03-27T12:11:29Z

As discussed in the Slack formula's channel, tried to cover most cases, so I ended with this matrix:

salt 2019.2, python 3 (installed from Saltstack's repo)

debian-9
ubuntu-18.04
centos-7

salt 2019.2, python 2 (installed from Fedora's repo)

fedora-29

salt 2018.3, python 2 (installed from Saltstack's repo)

opensuse-42.3
debian-8
ubuntu-16.04
centos-6

salt 2018.3, python 2 (installed from Fedora's repo)

forced-fedora-28: need to force the version, otherwise, dnf installs 2019.2 from the updates repo

salt 2017.7, python 2 (installed from Saltstack's repo)

debian-8
ubuntu-16.04
centos-6

Added tests to verify the package version in all cases, and split the tests dirs for the three salt versions, so we can now test config correctly for them.

Gemfile

myii · 2019-03-27T15:00:03Z

kitchen.yml

+driver:
+  name: docker
+
+driver_config:


As @dafyddj mentioned in Slack, lines 4 and 5 here can be removed, so that we end up with:

driver: name: docker use_sudo: false privileged: true provision_command: mkdir -p /run/sshd run_command: /lib/systemd/systemd

The same is true for lower down under platforms: too.

myii · 2019-03-27T23:34:07Z

kitchen.yml

+  # Previous distro, oldest salt, previous python
+  - name: debian-8-2017-7-py2
+    driver_config:
+      image: saltstack/au-debian-8:ci-2017.7-py2


image: registry.gitlab.com/dafyddj/debian-salt:8-2017.7

myii · 2019-03-28T20:49:50Z

@javierbertoli There's a strange problem: most images appear to end up with a Salt version that doesn't match the image that is being pulled. Based upon searching the log for sweet!. For example:

salt-debian-9-2019-2-py3 reports as 2018.3.3.
salt-ubuntu-1804-2019-2-py3 => 2018.3.3.
salt-centos-7-2019-2-py3 => 2018.3.3.
Etc.

javierbertoli · 2019-03-28T21:35:36Z

@myii interesting, it seems we have a few issues here:

The docker images don't ship what they say. On a freshly pulled image (expected salt 2019.2, python3):

$ docker run -ti saltstack/au-debian-9:ci-2019.2-py3 /bin/bash
root@be157c63728b:/# salt --version
salt 2018.3.3 (Oxygen)

root@be157c63728b:/# head -2 /usr/bin/salt
#!/usr/bin/python

root@be157c63728b:/# python --version
Python 2.7.13

This line, which I was aware that needed to be changed on suite-by-suite basis, is failing (as you mention) because running salt's bootstrapping script on the container above effectively updates it to latest and it's not happening on the kitchen-ci runs.
I wonder if these locale issues are causing the installer to fail.
I took a quick look at the kitchen-salt installer file and, comparing the lines on it with the output of the debian run logs it seems that the installer script is not being executed.

Use py3 for latest salt version Use Python 2 on old distros Update Gemfile Updated centos-6 run_command, as it does not use systemd Added tests to verify correct package versions for all cases tested

myii · 2019-03-31T23:34:56Z

@daks @aboe76 This is a fantastic effort from @javierbertoli. It would be great to get feedback from both of you, so that we can confirm this is how we want to proceed for this and other formulas as well.

@javierbertoli As mentioned on Slack, just one slight discrepancy in the log:

v2019-2-py3-centos-7 is showing as Python 2.

myii · 2019-04-01T01:46:54Z

@javierbertoli One request: we've run into bugs with salt-ssh, such as this one. Would it be possible to have one of the instances be a salt-ssh instance?

kitchen.yml

daks · 2019-04-01T10:41:53Z

Looks good to me, good job @javierbertoli :)

To match the one used by the bootstrapper. Otherwhise, we end up with two repos configured

javierbertoli · 2019-04-01T13:23:00Z

@javierbertoli As mentioned on Slack, just one slight discrepancy in the log:

v2019-2-py3-centos-7 is showing as Python 2.

Fixed. There's were three issues that caused this:

the salt's bootstrapper, at the begining of the kitchen run was running with default parameters and therefore installing python2 version of salt packages
when adding the repo, the repo name and file (saltstack-repo) were different than the ones installed by the bootstrapper (saltstack). Therefore, we ended up with two repos, one for python3 (the desired one) and one for python2. The same was happenening before for suse, that's why I added that weird repo name there.
And finally, when the state checked for the package, in the case of 2 (2 repos), it either found the package installed and did nothing (leaving a python2 packag) or, when fixing the repo, found inconsistent yum metadata and failed.

The changes pushed should fix all that. Bear in mind that these are weird situations caused by the kitchen-salt bootstrapping of salt to test formulas, and a salt formula trying to install salt in an instance with salt already installed on the bootstrapper. Too many hands in one plate 😋

@javierbertoli One request: we've run into bugs with salt-ssh, such as this one. Would it be possible to have one of the instances be a salt-ssh instance?

It's possible. Never tried salt-ssh in kitchen, but I think is doable. Any suggestion of what to test? Or else, we can submit another PR to address that?

myii · 2019-04-01T13:40:55Z

@javierbertoli As discussed on Slack just now, we'll leave this salt-ssh for a future PR. I'm happy with this PR to be merged, thanks again for the dedication! The folks at Travis CI are probably breathing a collective sigh of relief that this PR testing is over!

aboe76 · 2019-04-01T19:35:39Z

@javierbertoli and @myii what a work!!!,

I can only see one issue coming up, soon, when you switch to opensuse leap 15, it will be python3...

javierbertoli · 2019-04-01T19:41:29Z

There's an opensuse/leap:15 image available, I can switch to/add it to the matrix if you want.

We discussed it with @myii on slack and, as none of us is an active opensuse user, we are not aware of which version to use/prefer. So I went with the 42.3 because I remember we used it somewhere in the formulas.

But if you have a preference/suggestion of which opensuse images to use, I think I can change it. Either in this PR or adding a new issues/PR for it.

aboe76 · 2019-04-01T19:59:08Z

@javierbertoli I would prefer the opensuse/leap 15 version over the 42.3 version because of several reasons:

opensuse leap and sles have the same base image with opensuse leap 15.
opensuse leap 15 has 36 months of support while the 42.3 version only lifes till june 30 2019.
https://en.opensuse.org/Lifetime

javierbertoli · 2019-04-01T22:42:42Z

@aboe76, tried to change the code to opensuse Leap 15, but hit an issue with inspec: they're not currently supporting it, only 13 & 42 and therefore the tests are skipped

  ↺  salt packages: should be installed (2 skipped)
     ↺  The `package` resource is not supported on your OS yet.
     ↺  The `package` resource is not supported on your OS yet.
  ↺  salt services: should be running (2 skipped)
     ↺  The `service` resource is not supported on your OS yet.
     ↺  The `service` resource is not supported on your OS yet.

All in all, given that opensuse is in the matrix and is being tested, I think that we can upgrade to opensuse-leap-15 in a couple of months, or submit a patch to inspec? wdyt?

aboe76 · 2019-04-02T05:04:47Z

Thanks for trying, you did an amazing job. Op di 2 apr. 2019 00:42 schreef Javier Bértoli <notifications@github.com>:

…

@aboe76 <https://github.com/aboe76>, tried to change the code to opensuse Leap 15, but hit an issue with inspec: they're not currently supporting it <https://github.com/inspec/inspec/blob/master/README.md#supported-os>, only 13 & 42 and therefore the tests are skipped ↺ salt packages: should be installed (2 skipped) ↺ The `package` resource is not supported on your OS yet. ↺ The `package` resource is not supported on your OS yet. ↺ salt services: should be running (2 skipped) ↺ The `service` resource is not supported on your OS yet. ↺ The `service` resource is not supported on your OS yet. All in all, given that opensuse is in the matrix and is being tested, I think that we can upgrade to opensuse-leap-15 in a couple of months, or submit a patch to inspec? wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#406 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABt51MGJF_a8R02AXDvMwTLXSFzuUJ_Wks5vcotkgaJpZM4cNpBU> .

myii · 2019-04-02T06:09:59Z

@javierbertoli Merged -- thanks for this excellent contribution.

To everyone else, your feedback and comments were extremely helpful and much appreciated.

* Based on: saltstack-formulas/salt-formula#406

myii reviewed Mar 27, 2019

View reviewed changes

Gemfile Outdated Show resolved Hide resolved

myii reviewed Mar 27, 2019

View reviewed changes

javierbertoli force-pushed the master branch from c1998a1 to cdc6d29 Compare March 31, 2019 14:09

javierbertoli added 2 commits March 31, 2019 11:44

Update the testing matrix to cover most distro+salt+python combinations

b2d760c

Use py3 for latest salt version Use Python 2 on old distros Update Gemfile Updated centos-6 run_command, as it does not use systemd Added tests to verify correct package versions for all cases tested

Fix test repo dir, try to get centos-6 working

f494b59

javierbertoli force-pushed the master branch from cdc6d29 to f494b59 Compare March 31, 2019 15:03

We don't need the daemons running at bootstrap time

9f9bd97

javierbertoli force-pushed the master branch from 4e7f843 to 9f9bd97 Compare March 31, 2019 15:30

javierbertoli added 2 commits March 31, 2019 13:02

Specify dependencies

2e1d4c4

Bootstrap using git, so versions don't collide at package install time.

5ffc1e5

javierbertoli force-pushed the master branch 4 times, most recently from aa1c637 to 8017fa1 Compare March 31, 2019 21:07

Try to make centos-6 work

79109b9

javierbertoli force-pushed the master branch from 83adc59 to 79109b9 Compare March 31, 2019 22:27

javierbertoli changed the title ~~WIP - Update the testing matrix, use new docker images~~ Update the testing matrix (5 distros, 2 python versions, 3 salt versions) Mar 31, 2019

myii requested review from aboe76 and daks March 31, 2019 23:12

daks reviewed Apr 1, 2019

View reviewed changes

kitchen.yml Show resolved Hide resolved

Changed Redhat's repo name and file

4231029

To match the one used by the bootstrapper. Otherwhise, we end up with two repos configured

myii approved these changes Apr 1, 2019

View reviewed changes

myii merged commit 683f455 into saltstack-formulas:master Apr 2, 2019

This was referenced Apr 13, 2019

test(inspec): switch tests to use Inspec saltstack-formulas/nginx-formula#220

Merged

Use platform-version specific testing (not just the default suite) saltstack-formulas/nginx-formula#226

Open

myii added a commit to myii/template-formula that referenced this pull request Apr 20, 2019

ci(kitchen+travis): implement new distro-python-salt_version matrix

fa8d364

* Based on: saltstack-formulas/salt-formula#406

myii added a commit to myii/template-formula that referenced this pull request Apr 20, 2019

ci(kitchen+travis): implement new distro-python-salt_version matrix

52cc46e

* Based on: saltstack-formulas/salt-formula#406

myii added a commit to myii/template-formula that referenced this pull request Apr 20, 2019

ci(kitchen+travis): implement new distro-python-salt_version matrix

bd4792d

* Based on: saltstack-formulas/salt-formula#406

myii mentioned this pull request Apr 20, 2019

ci(kitchen+travis): implement new distro-python-salt_version matrix saltstack-formulas/template-formula#89

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the testing matrix (5 distros, 2 python versions, 3 salt versions) #406

Update the testing matrix (5 distros, 2 python versions, 3 salt versions) #406

javierbertoli commented Mar 27, 2019 •

edited

Loading

myii Mar 27, 2019

dafyddj Mar 27, 2019

myii Mar 27, 2019

myii commented Mar 28, 2019

javierbertoli commented Mar 28, 2019

myii commented Mar 31, 2019

myii commented Apr 1, 2019

daks commented Apr 1, 2019

javierbertoli commented Apr 1, 2019

myii commented Apr 1, 2019

aboe76 commented Apr 1, 2019

javierbertoli commented Apr 1, 2019

aboe76 commented Apr 1, 2019

javierbertoli commented Apr 1, 2019

aboe76 commented Apr 2, 2019 via email

myii commented Apr 2, 2019

Update the testing matrix (5 distros, 2 python versions, 3 salt versions) #406

Update the testing matrix (5 distros, 2 python versions, 3 salt versions) #406

Conversation

javierbertoli commented Mar 27, 2019 • edited Loading

salt 2019.2, python 3 (installed from Saltstack's repo)

salt 2019.2, python 2 (installed from Fedora's repo)

salt 2018.3, python 2 (installed from Saltstack's repo)

salt 2018.3, python 2 (installed from Fedora's repo)

salt 2017.7, python 2 (installed from Saltstack's repo)

myii Mar 27, 2019

Choose a reason for hiding this comment

dafyddj Mar 27, 2019

Choose a reason for hiding this comment

myii Mar 27, 2019

Choose a reason for hiding this comment

myii commented Mar 28, 2019

javierbertoli commented Mar 28, 2019

myii commented Mar 31, 2019

myii commented Apr 1, 2019

daks commented Apr 1, 2019

javierbertoli commented Apr 1, 2019

myii commented Apr 1, 2019

aboe76 commented Apr 1, 2019

javierbertoli commented Apr 1, 2019

aboe76 commented Apr 1, 2019

javierbertoli commented Apr 1, 2019

aboe76 commented Apr 2, 2019 via email

myii commented Apr 2, 2019

javierbertoli commented Mar 27, 2019 •

edited

Loading