Skip to content
This repository has been archived by the owner on May 6, 2020. It is now read-only.

qemu: Add qemu-cc #34

Closed
wants to merge 1 commit into from
Closed

qemu: Add qemu-cc #34

wants to merge 1 commit into from

Conversation

erick0z
Copy link

@erick0z erick0z commented Aug 3, 2017

This commit adds the OBS files to build qemu with Q35 type support.

Fixes #25

@erick0z
Copy link
Author

erick0z commented Aug 3, 2017

@gorozco1 May I have a do-not-merge tag for this?

@gorozco1
Copy link

gorozco1 commented Aug 3, 2017

@erick0z done

Copy link

@gorozco1 gorozco1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After Arch meeting the name chosen for the new qemu is qemu-cc please change.

@erick0z erick0z changed the title Add qemu-q35 Add qemu-cc Aug 4, 2017
@erick0z erick0z changed the title Add qemu-cc qemu: Add qemu-cc Aug 4, 2017
Copy link

@gorozco1 gorozco1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general OK, but we need some changes.

@@ -0,0 +1,5 @@
qemu-cc (2.9.0) stable; urgency=medium

* Initial version introducing Q35 type
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And removal of pc-lite

@@ -0,0 +1 @@
setBadness('arch-dependent-file-in-usr-share', 0)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null at the end of the file, can you fix this please?

@@ -19,6 +19,7 @@ short_hashtag="${hash_tag:0:7}"
[ -z "$hash_tag" ] && hash_tag=$VERSION || :

OBS_PUSH=${OBS_PUSH:-false}
NON_STAGING=${NON_STAGING:-false}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better call the variable STAGING for clarity. And always set to true

-e '/Requires: linux-container*/d' -i cc-runtime.spec
}

if [ "$NON_STAGING" == true ]; then
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just change a little bit the logic with a variable called STAGING

-e "s/@HASH_TAG@/$short_hashtag/g;" \
-e "s/@cc_proxy_version@/$proxy_obs_ubuntu_version/" \
-e "s/@cc_shim_version@/$shim_obs_ubuntu_version/" \
-e "s/@qemu_lite_version@/$qemu_lite_obs_ubuntu_version/" \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we will need to remove the dependency of qemu-lite or change the dependency for qemu-cc


%build
export LANG=C
%configure --prefix=/usr --disable-tools --disable-libssh2 --disable-tcmalloc --disable-glusterfs \

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks neat, but makes maintenance more difficult: could you split this so each option is put on a separate line:

--option-1 \
--option-2 \
  :


%files data
%defattr(-,root,root,-)
/usr/share/qemu/acpi-dsdt.aml

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks wrong to me - /usr/share/qemu/ will conflict with a distro-packaged version of qemu so you need to add a --libdir=, --libexecdir= and --datadir= as is used in qemu-lite.spec.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we don't want conflicts with the packaged qemu. Fixing...

Docs for the qemu-cc package.

%prep
ls -la /home/abuild/rpmbuild/SOURCES/

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed or is it some stray debug?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, that's my bad, thanks!


Requires: qemu-cc-bin
Requires: qemu-cc-data
Requires: qemu-cc-doc

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't necessarily wrong, but why are we now packaging the docs when we didn't in qemu-lite.spec?

@erick0z
Copy link
Author

erick0z commented Aug 7, 2017

@jodh-intel Thanks for the review, I've addressed the comments.

@gorozco1
Copy link

gorozco1 commented Aug 7, 2017

@erick0z please also update the README.rst to list qemu-cc on Included Components list

%defattr(-,root,root,-)
/usr/bin/qemu-cc-ga
/usr/bin/qemu-cc-system-x86_64
/usr/bin/virtfs-proxy-helper

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still going to conflict with a distro-packaged version of qemu. Please see how this is handled in qemu-lite.spec in the %install and %files sections.

(Aside: this file is quite different to qemu-lite.spec which makes comparison difficult).

BuildRequires : m4
BuildRequires : numactl-devel
BuildRequires : python-devel
BuildRequires : zlib

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The qemu-lite.spec specifies this as:

BuildRequires : zlib-devel

... which seems "more correct" since that will pull in the zlib package.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

BuildRequires : libtool-ltdl-devel
BuildRequires : libtool
BuildRequires : m4
BuildRequires : numactl-devel

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qemu-lite.spec has a test for SuSE here and adds libnuma-devel for that distro. Was that code dropped on purpose?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

--datadir=/usr/share/qemu-cc \
--libdir=/usr/lib64/qemu-cc \
--libexecdir=/usr/libexec/qemu-cc \
--disable-tools \

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few differences here between qemu-lite.spec. Please can you check and compare and either fix, or add a comment in the commit message explaining why each feature has been enabled/disabled compared to qemu-lite.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I took the configure flags from the travis setup (https://github.com/clearcontainers/tests/blob/master/.ci/setup_env_ubuntu.sh), so the package build remains the same as from-source build. I don't know the whole story of why some flags are enabled/disabled.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've got to be careful here for various reasons...

Here's how qemu-lite is built in clr:

That is almost identical to qemu-lite.spec except that that spec file adds two extra options:

$ comm -3 clr-qemu-lite.configure qemu-lite.spec 
        --disable-docs
        --disable-static

It should be safe to add those two to qemu-cc.spec.

Comparing qemu-lite.spec and setup_env_ubuntu.sh, it looks like setup_env_ubuntu.sh is missing:

--disable-uuid (I think this is valid as we always specify a value to `-uuid`, so qemu doesn't need to generate one).
--enable-kvm (this is admittedly the default)
--enable-vhost-net (needed!)
--disable-qom-cast-debug

Also -- oddly -- setup_ubuntu_env.sh adds the following...

--disable-cap-ng
--disable-attr
--enable-trace-backend=nop

I had thought that those disabled options were required for 9p. @grahamwhaley - do you have any input on that? I'm also not sure why it is explicitly enabling a trace backend, unless that effectively "disables" tracing which is enabled by default (the name suggests it might).

Finally, setup_ubuntu_env.sh doesn't specify any CFLAGS. That's fine for a CI build, but we need them for packaged builds for security reasons.

So, in summary, I think you need to remove the following from qemu-cc.spec:

--disable-attr
--disable-cap-ng

... add the following to qemu-cc.spec:

--disable-docs
--disable-qom-cast-debug
--disable-static
--disable-strip
--disable-uuid
--enable-attr
--enable-kvm
--enable-vhost-net
--enable-virtfs
--extra-cflags="-fno-semantic-interposition -O3 -falign-functions=32 -D_FORTIFY_SOURCE=2 -fPIE"
--extra-ldflags="-pie"

For the cflags options, see:

@sboeuf, @mcastelino, @grahamwhaley, @anthonyzxu - please can you review this?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've raised clearcontainers/tests#339 for further discussion on the problem of keeping the qemu* build options in sync "everywhere".

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jodh-intel nice!

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some discussion on the public irc channel, we've decided it makes sense to land this first, then at at later date rework all scripts/config files that need qemu options to call the script on clearcontainers/tests#340.

As a mimimum, this PR needs to add --enable-vhost-net. The other options listed above either should be fine to leave out for now (will not compromise functionality). The issue of the security options probably needs further discussion on clearcontainers/tests#340 as we're not building with those options anywhere afaik.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note the attr and cap-ng options seem to be related to virtfs-proxy-helper(1).

See: clearcontainers/tests@d4fba34 which is part of clearcontainers/tests#340.

Requires: qemu-cc-bin
Requires: qemu-cc-data

BuildRequires : automake

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qemu-lite.spec adds a BuildRequires on gcc-c++. Is that needed still?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I see, those changes were introduced recently for the opensuse build.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here #37

make install DESTDIR=%{buildroot} PREFIX=/usr
mv %{buildroot}/usr/bin/qemu-system-x86_64 %{buildroot}/usr/bin/qemu-cc-system-x86_64
mv %{buildroot}/usr/bin/qemu-ga %{buildroot}/usr/bin/qemu-cc-ga
mv %{buildroot}/usr/bin/virtfs-proxy-helper %{buildroot}/usr/bin/virtfs-cc-proxy-helper

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the explicitness of this, but my worry is that if qemu introduce any new binaries, this package will end up trying to overwrite the distro-packaged version. That was the rational for using a loop in qemu-lite.spec:

Note that if that technique is "re-adopted", it should also be used in the debian.rules file.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the explicitness too :). In the case of a new binary, the build system will notice and complain about it because it has to be listed below the %files directive. Anyway, I'm ok with whatever approach is adopted. @jodh-intel @gorozco1

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, if it's guaranteed to complain in that scenario, that's fine with me.

--datadir=/usr/share/qemu-cc \
--libdir=/usr/lib64/qemu-cc \
--libexecdir=/usr/libexec/qemu-cc \
--disable-tools \

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've got to be careful here for various reasons...

Here's how qemu-lite is built in clr:

That is almost identical to qemu-lite.spec except that that spec file adds two extra options:

$ comm -3 clr-qemu-lite.configure qemu-lite.spec 
        --disable-docs
        --disable-static

It should be safe to add those two to qemu-cc.spec.

Comparing qemu-lite.spec and setup_env_ubuntu.sh, it looks like setup_env_ubuntu.sh is missing:

--disable-uuid (I think this is valid as we always specify a value to `-uuid`, so qemu doesn't need to generate one).
--enable-kvm (this is admittedly the default)
--enable-vhost-net (needed!)
--disable-qom-cast-debug

Also -- oddly -- setup_ubuntu_env.sh adds the following...

--disable-cap-ng
--disable-attr
--enable-trace-backend=nop

I had thought that those disabled options were required for 9p. @grahamwhaley - do you have any input on that? I'm also not sure why it is explicitly enabling a trace backend, unless that effectively "disables" tracing which is enabled by default (the name suggests it might).

Finally, setup_ubuntu_env.sh doesn't specify any CFLAGS. That's fine for a CI build, but we need them for packaged builds for security reasons.

So, in summary, I think you need to remove the following from qemu-cc.spec:

--disable-attr
--disable-cap-ng

... add the following to qemu-cc.spec:

--disable-docs
--disable-qom-cast-debug
--disable-static
--disable-strip
--disable-uuid
--enable-attr
--enable-kvm
--enable-vhost-net
--enable-virtfs
--extra-cflags="-fno-semantic-interposition -O3 -falign-functions=32 -D_FORTIFY_SOURCE=2 -fPIE"
--extra-ldflags="-pie"

For the cflags options, see:

@sboeuf, @mcastelino, @grahamwhaley, @anthonyzxu - please can you review this?

@erick0z
Copy link
Author

erick0z commented Aug 9, 2017

@gorozco1 Can you remove the do-not-merge label? This PR is ready to be merged

@gorozco1
Copy link

@mcastelino could you mention the firmware that qemu-cc needs to include in the package.

@grahamwhaley
Copy link
Contributor

Hi @gorozco1 I believe it is 'bios.bin', or very similar. It depends on the machine type being used I think and the config. I ran into this when testing a hand installed q35 machine type with BIOS on a host that had no qemu :-)
I think the key QEMU source reference is here:
https://git.qemu.org/http://git.qemu.org/qemu.git/?p=qemu.git;a=blob;f=hw/i386/pc_sysfw.c;h=6b183747fcea51facffe8f94117bf2a421522316;hb=HEAD#l39

iirc, if we do not install the binary in our default search dir (such as /usr/share/qemu-lite/qemu or similar), then our qemu will pick up the system qemu installed bios file if it exists (such as from /usr/share/qemu). That will most likely work - but, we really should be installing our own version and to match the version of qemu we are using ;-)

@gorozco1
Copy link

Currently this is PR is under testing.

@gorozco1
Copy link

I just confirmed with @chavafg that the package include the firmware and the tests is getting it without any problems 👍

@chavafg
Copy link

chavafg commented Aug 11, 2017

There were issues with CRI-O tests when running with qemu-cc, so this should not be merged for Beta release.

@mcastelino
Copy link

@gorozco1 we need the following for our use of PC

qemu/pc-bios/bios-256k.bin
qemu/pc-bios/kvmvapic.bin
qemu/pc-bios/linuxboot_dma.bin

@anthonyzxu Do we need any other firmware.

@gorozco1
Copy link

@mcastelino I am almost sure we package those with qemu-cc I check with @chavafg and seems that we are packaging those. @chavafg could you list the package contents please?

@anthonyzxu
Copy link

only
qemu/pc-bios/bios-256k.bin
is needed

qemu/pc-bios/kvmvapic.bin
qemu/pc-bios/linuxboot_dma.bin

are not needed.

@chavafg
Copy link

chavafg commented Aug 11, 2017

We provide all those that @mcastelino commented:

/usr/share/qemu-cc/qemu/bios-256k.bin
/usr/share/qemu-cc/qemu/kvmvapic.bin
/usr/share/qemu-cc/qemu/linuxboot_dma.bin

@chavafg
Copy link

chavafg commented Nov 14, 2017

@sboeuf here are the console logs:
http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/clear-containers-tests-azure-ubuntu-17-04-PR/199/consoleText
I tried to look at the cc-runtime logs, but didnt find any error there.

This commit adds the OBS files to build qemu with Q35 type support.
It also removes qemu-lite as dependency of the runtime.

Fixes #25

Signed-off-by: Erick Cardona <erick.cardona.ruiz@intel.com>
@erick0z
Copy link
Author

erick0z commented Nov 14, 2017

@sboeuf @chavafg Just updated this PR. New packages can be found here: https://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/. I'll let you know when we can retrieve these from clearlinux.org.

@sboeuf
Copy link

sboeuf commented Nov 14, 2017

Thanks @erick0z !

@amshinde
Copy link

@chavafg Can we compare the versions of devicemapper on ubuntu and fedora? Maybe there were some fixes pushed.

@chavafg
Copy link

chavafg commented Nov 22, 2017

@amshinde the Fedora version is newer than the Ubuntu one.

Ubuntu:

fuentess@singlevm:~$ pkg-config --modversion devmapper
1.02.110
fuentess@singlevm:~$ pkg-config --modversion devmapper-event
1.02.110
fuentess@singlevm:~$ dpkg -l | grep devmapper
ii  libdevmapper-dev:amd64           2:1.02.110-1ubuntu10                        amd64        Linux Kernel Device Mapper header files
ii  libdevmapper-event1.02.1:amd64   2:1.02.110-1ubuntu10                        amd64        Linux Kernel Device Mapper event support library
ii  libdevmapper1.02.1:amd64         2:1.02.110-1ubuntu10                        amd64        Linux Kernel Device Mapper userspace library

Fedora:

[cc-tester@singlevm ~]$ rpm -qa | grep device-mapper
device-mapper-1.02.136-3.fc25.x86_64
device-mapper-libs-1.02.136-3.fc25.x86_64
device-mapper-devel-1.02.136-3.fc25.x86_64

[cc-tester@singlevm ~]$ pkg-config --modversion devmapper
1.02.136

@chavafg
Copy link

chavafg commented Nov 22, 2017

Will try to update ubuntu's version and see if the problem goes away.

@sboeuf
Copy link

sboeuf commented Nov 22, 2017

@chavafg I think you mean you will try to downgrade devicemapper version for Ubuntu in order to check if the proble goes away, right ?

@chavafg
Copy link

chavafg commented Nov 23, 2017

@sboeuf no, ubuntu version is lower than the fedora one (where the tests work well), so I tried by updating devicemapper to the same version that fedora has.

But after updating device-mapper to 1.02.136 in Ubuntu, I still have the same errors.

@amshinde
Copy link

@fuentess Have you tried this with the latest agent/image. I reran the crio tests locally today on Ubuntu 16.04 and saw that the tests are passing. Can you retest?

@egernst
Copy link

egernst commented Feb 15, 2018

This sounds like the same bug as clearcontainers/runtime#910

Once the virtio-scsi lands, perhaps we can test here. Seems virtio-blk support is buggy when moving to later QEMU versions,.

@egernst
Copy link

egernst commented Feb 15, 2018

related PR for scsi: containers/virtcontainers#584

@egernst
Copy link

egernst commented Feb 15, 2018

@sboeuf ^^ see issue above for logs regarding the mount failures observed; likely the same which is observed during our crio tests with qemu-cc.

--datadir=/usr/share/qemu-cc \
--libdir=/usr/lib64/qemu-cc \
--libexecdir=/usr/libexec/qemu-cc \
--disable-tools \

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sboeuf
Copy link

sboeuf commented Feb 28, 2018

@chavafg could you please give this a new try after clearcontainers/runtime#1037 has been merged. My understanding is that we got blocked all this time because we were using virtio-blk and there is a race condition issue not resolved for latest Qemu versions (> 2.9).
But now this should pass and if it does as we expect, we'll be able to move to Qemu 2.9 or 2.10 !

@sboeuf
Copy link

sboeuf commented Mar 2, 2018

@chavafg clearcontainers/runtime#1037 has been merged 😄, it's time to check if switching to Qemu 2.9 can be done !

@egernst
Copy link

egernst commented Mar 3, 2018

2.10 or greater or bust!

@sboeuf
Copy link

sboeuf commented Mar 3, 2018

@egernst of course ! 2.10 or greater 😄

@chavafg
Copy link

chavafg commented Mar 5, 2018

verifying today... will post back when having the results.

@sboeuf
Copy link

sboeuf commented Mar 5, 2018

Thank you @chavafg

@chavafg
Copy link

chavafg commented Mar 6, 2018

So I have tested using qemu 2.10 and virtio-scsi on a local VM using ccloudvm. Ran all tests in a ten iteration loop and they passed correctly.

But I also tried using Azure VMs and I am not able to run containers.
The cc-runtime hangs on the create command. It launches a qemu process, but I do not see any errors in the cc-runtime or cc-proxy logs. The shim does not log anything.
As you can see in clearcontainers/tests#942 and in http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/clear-containers-tests-azure-ubuntu-16-04-PR/553/consoleFull, the tests failed on timeouts.

Any Idea of what might be wrong in the azure machines?

@sboeuf
Copy link

sboeuf commented Mar 6, 2018

@chavafg does Qemu 2.10 need specific libraries or a specific kernel config ? Something seems to be missing on the host environment.

@chavafg
Copy link

chavafg commented Mar 6, 2018

@sboeuf not sure, because I could create containers using qemu 2.10 and virtio-blk

@sboeuf
Copy link

sboeuf commented Mar 6, 2018

@chavafg can you go on Azure and verify that you can manually start a VM with Qemu 2.10 ? We need to identify if the issue comes from some misconfiguration of the environment, or if it comes from the way Qemu 2.10 is used by Clear Containers.

@mcastelino
Copy link

@chavafg @sboeuf have we added the virtio workaround for the kvm instruction emulation issue to virtio-scsi. We need this workaround till the kvm fix makes its way into the distros

@mcastelino
Copy link

mcastelino commented Mar 6, 2018

@sboeuf
Copy link

sboeuf commented Mar 6, 2018

@mcastelino ok but this is a patch for the host kernel, which means this is something we don't really have control on. What is the workaround that we should use in the meantime ?

egernst pushed a commit to egernst/cc-packaging that referenced this pull request Aug 14, 2018
Fix qemu build in distros with new glibc versions.
@egernst egernst closed this Mar 26, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.