Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move integration tests to Ubuntu Focal #2486

Conversation

RenaudWasTaken
Copy link
Contributor

Hello!

Following up on the requests of #2229
Here I moved all the integration tests from busybox to debian. This involved the following three operations:

  • find and replace all occurrences of busybox by debian in the following three contexts:
    • teardown_busybox, setup_busybox
    • BUSYBOX_BUNDLE
    • runc run ./work-dir test_busybox
  • Generate the debian spec using runc rather than umoci
    • This had the implication that we needed to update the hooks test to set the container root to read-write (by default runc generates a spec with a read-only root)
  • Update some of the error messages that were slightly different with Debian (e.g: Permission denied) and error codes

/cc @cyphar @AkihiroSuda

Thanks for reviewing!

Signed-off-by: Renaud Gaubert rgaubert@nvidia.com

@RenaudWasTaken
Copy link
Contributor Author

Hmm, I wasn't expecting debian not to have ps in the base image...

@cyphar
Copy link
Member

cyphar commented Jun 24, 2020

This has bitten me in the past as well. And now is the part where I pitch openSUSE 😉 -- the opensuse/leap:15.1 image is smaller than the Debian one and has both ldconfig and ps in the base image.

@RenaudWasTaken
Copy link
Contributor Author

Hmm, I'll look into using the opensuse/leap image then!

@cyphar
Copy link
Member

cyphar commented Jun 24, 2020

One other thing you'll need to update is the Go "integration" tests in libcontainer/integration -- they use the busybox image as well. It also wouldn't hurt to drop the hello image as well, which is also a slightly questionable thing we still have hanging around.

@RenaudWasTaken
Copy link
Contributor Author

RenaudWasTaken commented Jun 24, 2020

Looks like opensuse works on all bats the integration tests.
I'm hitting a small issue in the go integration tests related to groups:

--- FAIL: TestAdditionalGroups (0.26s)
utils_test.go:55: exec_test.go:487: unexpected error: container_linux.go:367: starting container process caused: setup user: Unable to find group plugdev

Will look into it a bit more tomorrow morning :) !

Update: Renaud re-discovered groups in Linux today....

It also wouldn't hurt to drop the hello image as well, which is also a slightly questionable thing we still have hanging around

I'm happy to drop that one :)

@cyphar
Copy link
Member

cyphar commented Jun 24, 2020

Ah, right. Yeah you'll just need to pick a different group to run the user as (openSUSE doesn't have a plugdev group).

@RenaudWasTaken RenaudWasTaken force-pushed the integration-tests-debian branch 6 times, most recently from e280b19 to 2436fc8 Compare June 25, 2020 02:41
@RenaudWasTaken
Copy link
Contributor Author

RenaudWasTaken commented Jun 25, 2020

Working through the last issues!

@cyphar it looks like I can't just rm -rf after a umoci unpack --rootless for the opensuse image. For some reason I need to chmod 755 the following dir: ./rootfs/var/lib/ca-certificates/".

Are you ok with that workaround until we investigate a bit more what is going on there?

e.g:

# sudo -HE -u rootless bash
$ opensuse="opensuse:3.11.6"
$ tmp=$(mktemp -d)
$ cd "$tmp"
$ skopeo copy docker://opensuse/leap:15.1 "oci:$opensuse"
$ umoci unpack --rootless --image "$opensuse" "./"
$ rm -rf ./rootfs
rm: cannot remove 'rootfs/var/lib/ca-certificates/pem/2fa87019.0': Permission denied
....
$ chmod -R 755 "./rootfs/var/lib/ca-certificates/"
$ rm -rf ./rootfs

@RenaudWasTaken RenaudWasTaken force-pushed the integration-tests-debian branch 8 times, most recently from 82d9aaa to d880b40 Compare June 26, 2020 01:44
@RenaudWasTaken RenaudWasTaken force-pushed the integration-tests-debian branch 5 times, most recently from 7e62848 to f31c27f Compare June 29, 2020 05:13
@cyphar
Copy link
Member

cyphar commented Jul 2, 2020

Yeah that's fine @RenaudWasTaken -- Tumbleweed is a rolling-release distribution.

@cyphar
Copy link
Member

cyphar commented Jul 2, 2020

@AkihiroSuda

Where is the list of the supported architectures? Couldn't find on google 😭

I only just found this out, but you can find it by looking at the tags tab on Docker Hub and you can see which architectures we build for here.

@AkihiroSuda
Copy link
Member

Will openSUSE support RISC-V and MIPS?

@RenaudWasTaken
Copy link
Contributor Author

Switched to tumbleweed

@RenaudWasTaken
Copy link
Contributor Author

RenaudWasTaken commented Jul 2, 2020

So umount is not part of tumbleweed!
Update: Nevermind, just need to update the test , it's called unmount

@RenaudWasTaken
Copy link
Contributor Author

It does seem like it's missing the mount command though....

@RenaudWasTaken
Copy link
Contributor Author

RenaudWasTaken commented Jul 10, 2020

I opened a request on the opensuse base image to add the mount command: https://build.opensuse.org/request/show/819842

A few alternatives that comes to mind if we can't use the base image (untested):

  • Docker in Docker to build a custom container before skopeo + umoci
  • skopeo + umoci + chroot (and a few mounts) + zypper

@RenaudWasTaken RenaudWasTaken force-pushed the integration-tests-debian branch 2 times, most recently from a4a2bbb to 66deae3 Compare July 30, 2020 05:37
@RenaudWasTaken
Copy link
Contributor Author

RenaudWasTaken commented Jul 30, 2020

  • Rebased
  • Moved to Ubuntu as it seems to be the only image that matches all our requirements:

For RISC-V and MIPS it seems like there are no official images.

@RenaudWasTaken RenaudWasTaken force-pushed the integration-tests-debian branch 4 times, most recently from 08ab4d2 to 59dc748 Compare August 9, 2020 08:07
@RenaudWasTaken
Copy link
Contributor Author

Sorry I hadn't noticed that the CI was failing, fixed!

Copy link
Member

@AkihiroSuda AkihiroSuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please squash commits and modify the PR description texts to reflect the reality. Also, what was the problem of using Debian? It's the best operating system for multi arch.

Copy link
Member

@AkihiroSuda AkihiroSuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please squash commits and modify the PR description texts to reflect the reality. Also, what was the problem of using Debian? It's the best operating system for multi arch.

Copy link
Member

@AkihiroSuda AkihiroSuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please squash commits and modify the PR description texts to reflect the reality. Also, what was the problem of using Debian? It's the best operating system for multi arch.

@AkihiroSuda
Copy link
Member

Lol github amplified my comment three times

@RenaudWasTaken
Copy link
Contributor Author

Also, what was the problem of using Debian? It's the best operating system for multi arch.

debian does not have ps in the base image.
I have a few thoughts on how we would get that utility in the image (e.g: Docker in Docker or chroot) but nothing very clean...

Signed-off-by: Renaud Gaubert <rgaubert@nvidia.com>
@AkihiroSuda
Copy link
Member

Why can't we just aptget it?

@RenaudWasTaken
Copy link
Contributor Author

RenaudWasTaken commented Aug 9, 2020

So the way the multi-arch script operates today is a simple:

skopeo copy docker://ubuntu:focal ...
umoci unpack --image ...

This works well in the Vagrant and Docker approach because it doesn't require spinning a container. If we wanted to apt-get a package inside that image, we have a few paths but they each seem to add a good amount of complexity on the CI:

  • Docker in Docker: docker build + skopeo copy on that image + umoci unpack
  • Some combination of wget procps package+ chroot + dpkg
  • A separate pipeline where we build a custom debian container

It's possible that I'm missing a completely obvious solution though :D

@AkihiroSuda AkihiroSuda changed the title Move integration tests to opensuse Move integration tests to Ubuntu Focal Aug 10, 2020
@kolyshkin
Copy link
Contributor

This is very obsoleted now.

@kolyshkin kolyshkin closed this Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants