Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distribute docker container to run ATH #238

Merged
merged 17 commits into from
Aug 16, 2017
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
// For ci.jenkins.io
// https://github.com/jenkins-infra/documentation/blob/master/ci.adoc

def mavenName = 'mvn'
def jdkName = 'jdk8'

node('highram&&docker') {
withEnv([
"JAVA_HOME=${tool jdkName}",
"PATH+MAVEN=${tool mavenName}/bin:${env.JAVA_HOME}/bin"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need these at all, if the build is done inside the image anyway?

]) {
changelog scm

sh 'docker build --build-arg=uid=$(id -u) --build-arg=gid=$(id -g) -t jenkins/ath src/main/resources/ath-container'
Copy link
Contributor

@scoheb scoheb Jan 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As @rtyler suggested, here is the code to make use of docker.build():

def uid = sh returnStdout: true, script: "id -u | tr '\n' ' '"
def gid = sh returnStdout: true, script: "id -g | tr '\n' ' '"

def buildArgs = "--build-arg=uid=${uid} --build-arg=gid=${gid} src/main/resources/ath-container"
docker.build('jenkins/ath', buildArgs)

String containerArgs = '-v /var/run/docker.sock:/var/run/docker.sock'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might need --group-add=${gid} to make sure docker.sock is writable. Unclear to me whether that is handled already by the other tricks.

docker.image('jenkins/ath').inside(containerArgs) {
sh '''
eval $(./vnc.sh)
./run.sh firefox latest -Dmaven.test.failure.ignore=true -DforkCount=1 -B
'''
}

junit 'target/surefire-reports/TEST-*.xml'
}
}
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ it's own sandboxed workspace.
* [Obtaining a report of plugins that were exercised](docs/EXERCISEDPLUGINSREPORTER.md)
* [Managing the versions of Jenkins and plugins](docs/SUT-VERSIONS.md)
* [Investigation](docs/INVESTIGATION.md)
* [Running tests in container](docs/DOCKER.md)
* Selecting tests based on plugins they cover (TODO)

### Writing tests
Expand Down
20 changes: 20 additions & 0 deletions docs/DOCKER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Running tests in docker container

Depending on the CI infrastructure setup one may need to run the ATH itself in a docker container with access to the host docker service, following a strategy similar to the one described in [this article](http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/). When this is needed, the docker fixtures should not be accessed through mapped ports on the docker host, but directly through their container IP and port since they are "sibling" containers to the ATH. To enable this, set the environment variable SHARED_DOCKER_SERVICE=true, and then the functions ipBound(n) and port(n) will just return the container's IP and port where the fixture can be accessed.

TO-DO: Instead of depending on setting this environment variable, in the future we could try to somehow automatically detect the situation.

Interactive shell:

$ docker build --build-arg=uid=$(id -u) --build-arg=gid=$(id -g) -t jenkins/ath src/main/resources/ath-container
$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/home/ath-user/ath --user ath-user jenkins/ath bash -c 'cd $HOME/ath && bash'
$ eval $(./vnc.sh)
$ ./run.sh firefox latest -Dmaven.test.failure.ignore=true -DforkCount=1 -B

Jenkinsfile:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems redundant. Anyone who cares can look up the Jenkinsfile for themselves.


... // Checkout ATH
sh 'docker build --build-arg=uid=$(id -u) --build-arg=gid=$(id -g) -t jenkins/ath src/main/resources/ath-container'
docker.image('jenkins/ath').inside {
sh 'eval $(./vnc.sh) && ./run.sh firefox latest -Dmaven.test.failure.ignore=true -DforkCount=1 -B'
}
12 changes: 0 additions & 12 deletions docs/FIXTURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,15 +76,3 @@ public class TestContainer extends DockerContainer {
```

In this case the docker file must be located at classpath and in `org/ame/fixture/test` instead of the default location.

## Running the ATH itself as a docker container
Depending on the CI infrastructure setup one may need to run the ATH itself in a docker container with access to the host docker service,
following a strategy similar to the one described in [this article](http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/).

When this is needed, the docker fixtures should not be accessed through mapped ports on the docker host, but directly
through their container IP and port since they are "sibling" containers to the ATH.

To enable this, set the environment variable `SHARED_DOCKER_SERVICE=true`, and then the functions ipBound(n) and port(n)
will just return the container's IP and port where the fixture can be accessed.

TO-DO: Instead of depending on setting this environment variable, in the future we could try to somehow automatically detect the situation.
44 changes: 44 additions & 0 deletions src/main/resources/ath-container/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Dockerfile to be used to run ATH itself.
#
# see docs/DOCKER.md for usage
#

FROM debian:sid
MAINTAINER ogondza@gmail.com

RUN apt-get -y update && \
apt-get install -y \
curl \
docker.io \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also just download a specific client executable.

imagemagick \
iptables \
firefox-esr \
maven \
openjdk-8-jdk \
vnc4server

# Allow injecting uid and git to match directory ownership
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sufficiently weird you had better document clearly why this is necessary.

ARG uid=1000
ENV uid $uid
ARG gid=1000
ENV gid $gid

# So it is owned by root and has the permissions vncserver seems to require:
RUN mkdir /tmp/.X11-unix && chmod 1777 /tmp/.X11-unix/

RUN groupadd ath-user -g $gid && \
useradd ath-user -u $uid -g $gid -m -d /home/ath-user

# TODO seems this can be picked up from the host, which is unwanted:
ENV XAUTHORITY /home/ath-user/.Xauthority

USER ath-user
RUN mkdir /home/ath-user/.vnc && (echo ath-user; echo ath-user) | vncpasswd /home/ath-user/.vnc/passwd
# Default content includes x-window-manager, which is not installed, plus other stuff we do not need (vncconfig, x-terminal-emulator, etc.):
RUN touch /home/ath-user/.vnc/xstartup && chmod a+x /home/ath-user/.vnc/xstartup

# Set SUID and SGID for docker binary so it can communicate with mapped socket its uid:gid we can not control. Alternative
# approach used for this is adding ath-user to the group of /var/run/docker.sock but that require root permission we do not
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hence --group-add. Maybe.

# have in ENTRYPOINT as the container is started as ath-user.
USER root
RUN chmod ug+s "$(which docker)"
4 changes: 3 additions & 1 deletion vnc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ fi

vncserver -kill $display
vncserver -geometry 1750x1250 $display > /dev/null
(vncviewer $display || vncviewer localhost$display) > /dev/null &
if command -v vncviewer >/dev/null 2>&1; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should expose port 9042 or whatever it is, so people can attach a desktop vncviewer to the container and watch the magic happen.

(vncviewer $display || vncviewer localhost$display) > /dev/null &
fi

echo export BROWSER_DISPLAY=$display