Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issues for spurious failures related to IPv6 problems in Travis CI's new 2017-Q4 image #47002

Closed
kennytm opened this issue Dec 25, 2017 · 2 comments
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@kennytm
Copy link
Member

kennytm commented Dec 25, 2017

Upstream issue: travis-ci/travis-ci#8891.

1. Network tests related to IPv6 are failing

Error first happens in #46692. Fixed in #44694 by using the older Travis image instead.

2. Docker failed to start due to "Address already in use"

The old image is reverted in #46924. It was still going well for ~5.5 hours (2 more PRs) after the PR is merged, but when bors try #46941, the jobs all immediately fail with

...
[00:01:45] Sending build context to Docker daemon  499.2kB
[00:01:45] Step 1/10 : FROM ubuntu:16.04
[00:01:45]  ---> 00fd29ccc6f1
[00:01:45] Step 2/10 : RUN apt-get update && apt-get install -y --no-install-recommends   clang   make   file   curl   ca-certificates   python2.7   git   cmake   sudo   bzip2   xz-utils   wget   libssl-dev   pkg-config
[00:01:45]  ---> Using cache
[00:01:45]  ---> f82100b9eb89
[00:01:45] Step 3/10 : COPY scripts/freebsd-toolchain.sh /tmp/
[00:01:45]  ---> Using cache
[00:01:45]  ---> 41afc339a9b0
[00:01:45] Step 4/10 : RUN /tmp/freebsd-toolchain.sh i686
[00:01:45]  ---> Running in 66ec201d700e
[00:01:45] Address already in use
[00:01:45] The command has failed after 5 attempts.

The failure is extremely quick and rapidly burnt out the entire tree. This has been fixed by #47000.

@kennytm kennytm added A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Dec 25, 2017
bors added a commit that referenced this issue Dec 25, 2017
Follow up to #46924, fix massive spurious failure when starting docker

It seems using `fe80::/64` causes `docker start` to fail with "Address already in use". Try to change to a unique local address range instead.

`fe80::/64` is a link-local address (similar to `169.254.0.0/16` in IPv4). Let's try to use a random "private network" address to see whether that fixes things.

cc #47002

r? @aidanhs
@kennytm
Copy link
Member Author

kennytm commented Dec 25, 2017

Timeline of the second bug:

  1. 2017-12-23T14:12:10Z — Revert #46694 (Temporarily use the old Travis image) #46924 rolled up into Rollup of 10 pull requests #46967
  2. 2017-12-23T15:17:33Z — It was noted that the roll up failed in Travis with "Address already in use" while building docker image. Revert #46694 (Temporarily use the old Travis image) #46924 is backed out from the rollup.
  3. 2017-12-25T10:07:15Z — Revert #46694 (Temporarily use the old Travis image) #46924 has been merged.
  4. (MIR borrowck: no "move occurs because X is not Copy` error #46949 and Update compiler_builtins #46971 are merged successfully afterwards).
  5. 2017-12-25T15:46:13Z — Re-do the FreeBSD cross-builds to use Clang and libc++. Fixes #44433 #46941 experienced the beginning of spurious failure.
  6. 2017-12-25T18:37:43Z — Error noted, suspecting to be caused by the IPv6 address. Fix attempt filed at Follow up to #46924, fix massive spurious failure when starting docker #47000.
  7. 2017-12-26T08:31:07Z — Follow up to #46924, fix massive spurious failure when starting docker #47000 is merged, and the bug no longer happens afterwards.

@kennytm
Copy link
Member Author

kennytm commented Feb 12, 2018

Since we are not seeing any more spurious errors related to this image for over a month, I consider the image is now stable and am closing this issue.

@kennytm kennytm closed this as completed Feb 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

1 participant