Investigate caching .git data on Travis/AppVeyor

[alexcrichton]

The rust-lang/rust repo itself takes awhile to clone but we somewhat mitigate that with --depth=1 clones. Our submodules, however, are much larger and unfortunately cannot be cloned with a --depth argument due to how the branches work. This typically means that cloning the LLVM repo takes quite a long time! Unfortunately this also increases our chances to network problems by requiring a lot of data to move over the network.

When playing around with CircleCI recently I found that they automatically cached git repository data which greatly sped up cloning the repository and checking out submodules. Overall it felt quite nifty! We should investigate to see if a similar strategy can apply to Travis and/or AppVeyor. I'm not personally familiar with how CircleCI's git caching works, so some investigation there would be needed (and comments if you're familiar with it would be most welcome!)

Overall I would expec this change to:

• Reduce timeouts when cloning LLVM because very little data would transfer over the network
• Speed up builds (slightly). It'd save ~10 minutes on AppVeyor and ~2 minutes on Travis from the look of today's timings.

Any help to implement this would be very much appreciated!

[sfackler]

On the Travis side, it should just be a matter of enumerating all of the directories you want: https://docs.travis-ci.com/user/caching/#Arbitrary-directories

[alexcrichton]

Yeah unfortunately I don't actually know what directories are cached here. Is it just .git? (or maybe more?)

If it's just .git then this would make a great PR!

[aidanhs]

I've made a stab in #40780. It's not as simple as caching the .git directory - by the time the script runs, you already have a branch checked out and it's not really clear how you'd move the objects around. It's also not as simple as just copying .git/modules - I'd be pretty paranoid about that breaking if someone updates the branch of a gitmodule, as I doubt git is robust at handling people messing around in there.

Instead, I've tried to go the route of using the --reference flag, which explicitly uses another repo as a cache. Unfortunately this isn't particularly well integrated with submodules, so I've had to do some workarounds.

As an aside, I'm a bit suspicious that depth: 1 (added in #35085) doesn't do anything useful. Here's the initial clone log from my travis build:

$ git clone --depth=1 https://github.com/rust-lang/rust.git rust-lang/rust
Cloning into 'rust-lang/rust'...
remote: Counting objects: 9861, done.
remote: Compressing objects: 100% (8740/8740), done.
remote: Total 9861 (delta 5307), reused 2301 (delta 1034), pack-reused 0
Receiving objects: 100% (9861/9861), 8.01 MiB | 10.89 MiB/s, done.
Resolving deltas: 100% (5307/5307), done.
$ cd rust-lang/rust
$ git fetch origin +refs/pull/40780/merge:
remote: Counting objects: 627478, done.
remote: Compressing objects: 100% (117810/117810), done.
remote: Total 627478 (delta 510690), reused 622389 (delta 505626), pack-reused 0
Receiving objects: 100% (627478/627478), 285.82 MiB | 24.97 MiB/s, done.
Resolving deltas: 100% (510690/510690), completed with 3765 local objects.
From https://github.com/rust-lang/rust
 * branch                  refs/pull/40780/merge -> FETCH_HEAD
$ git checkout -qf FETCH_HEAD

Seems like it has to fetch the whole thing anyway in order to get my branch? Maybe worth raising an issue on the travis issue tracker to ask if this is expected?

[aidanhs]

FWIW, CircleCI is a bit vague about how their git caching works:

https://circleci.com/docs/1.0/how-cache-works/#git-cache

There’s another type of caching that CircleCI does. This is to cache the Git repository of your project. This feature allows for speedier git clones of your project. Control of this cache isn’t user accessible and shouldn’t need to be. In rare scenarios, if something seems to be wrong with your source cache (for example, certain files are in your repo that shouldn’t be), please contact support.

This opacity combined with complaints about the caching ([1], [2]) is not reassuring. I could hazard some guesses at how they're doing it, but it's likely going to be more complicated and therefore less reliable than an approach tailored to the rust repo.

[nagisa]

We used to do this on buildbot didn't we? IIRC corrupted .git was pretty common occurrence.

[aidanhs]

I assume you're referring to #34595, which looks like it happens because the cache may contain a partially cloned/corrupt git submodule. This was a problem even without caching (network failure then retry is unhappy because of this same corrupted state) and was seemingly resolved by deinit before update (#39055) - the same fix may have fixed the buildbots (had they still been around).

That said, I can think of two approaches for validating before caching off the top of my head (the second problem with the buildbots, aside from not doing deinit, was caching continuously rather than only when the cache was valid) - if we see issues (or you want a more paranoid approach to begin with) I'll be able to put something in place fairly quickly.

[alexcrichton]

Thanks for the investigation @aidanhs! I'll take a look at #40780 soon.

Also yeah @nagisa I'd want to be careful about a solution here. Bad caching can cause unending problems, so I'd want to make sure we're always in a situation where the tool we're caching is relatively robust to odd cache entries. For example docker I trust to do the right thing with caching (and naturally rebuilding when necessary), and same with sccache. I was hoping that this would be solved by just persisting some git database that's resilient to naturally being updated and having old cache entries, but we'll see soon!

[notriddle]

Call me stupid (since I probably am missing something obvious), but why do Travis CI and Circle CI workers even download the commit history? GitHub has the "download a tarball from https://github.com/rust-lang/rust/archive/master.tar.gz" option, which is probably way faster.

[aidanhs]

@notriddle if you already have a repo of N commits, downloading the N+1th commit with git pull will be a few KB. Downloading the entire repo every time is a constant ~6MB. Regardless, we need some kind of git clone for submodules to work (as the commitish they're pinned to lives in the .git directory and downloaded .tar.gz contain neither .git nor checked out submodules).

That said, Travis definitely has something sub-optimal with cloning - it should be doing a shallow clone of the branch, but instead it does a shallow clone of master and then a full clone of PR branches (defeating the point of the shallow clone).

[aidanhs]

After a bit of a shaky start (merge completed successfully, then subsequent merges failed because appveyor caching was broken) just the appveyor part was rolled back and so just travis builds are currently using the new repo caching and it seems to be working ok.

In the middle of trying to fix appveyor last night, I realised that there is a way for me to test appveyor - fork rust, then comment out all CI that actually does any rust compilation etc. That way I'll be able to just test the caching. I'll be back soon with a tested PR for appveyor.

[alexcrichton]

Thanksd for the continuing investigation @aidanhs!

[alexcrichton]

(oops didn't mean to close)

[aidanhs]

Unfortunately, I was completely unable to reproduce the cache restore failure on appveyor, despite trying a number of times. However, after reviewing the logs again, I don't actually think the cache restore failure was the issue, I think it was a different buggy part of the appveyor.yml. There's a new PR to re-enable appveyor at #41075.

---

Cache aside, as part of this issue I do think it's worth looking into this sequence of commands on travis:

$ git clone --depth=1 https://github.com/rust-lang/rust.git rust-lang/rust
Cloning into 'rust-lang/rust'...
remote: Counting objects: 10010, done.
remote: Compressing objects: 100% (8825/8825), done.
remote: Total 10010 (delta 5369), reused 2433 (delta 1096), pack-reused 0
Receiving objects: 100% (10010/10010), 8.04 MiB | 10.93 MiB/s, done.
Resolving deltas: 100% (5369/5369), done.
$ cd rust-lang/rust
$ git fetch origin +refs/pull/41074/merge:
remote: Counting objects: 630073, done.
remote: Compressing objects: 100% (118309/118309), done.
remote: Total 630073 (delta 512723), reused 625020 (delta 507693), pack-reused 0
Receiving objects: 100% (630073/630073), 286.78 MiB | 25.92 MiB/s, done.
Resolving deltas: 100% (512723/512723), completed with 3750 local objects.
From https://github.com/rust-lang/rust
 * branch                  refs/pull/41074/merge -> FETCH_HEAD

I described this above:

That said, Travis definitely has something sub-optimal with cloning - it should be doing a shallow clone of the branch, but instead it does a shallow clone of master and then a full clone of PR branches (defeating the point of the shallow clone).

The extra time spent here makes the cache a little less effective (40s wasted), and consumes about 60-75% of the time on the no-op PR builds - that's ~ 35x45s across Linux no-op builds and ~ 5x190s across OSX no-op builds, totalling 40-45mins of dead time per PR push! Seems massively wasteful, even if it is in parallel.

I've stumbled across something that looks like the ideal solution - appveyor lets you implement a custom clone_script, we just need to convince travis to implement it or find a workaround to do something equivalent.

[aidanhs]

Ok, the inefficiency in travis has been spotted before - travis-ci/travis-ci#6183, travis-ci/travis-build#747. I guess it just needs resurrecting and fixing. Appveyor seems to do it like so:

git clone -q --depth=1 --branch=auto https://github.com/rust-lang/rust.git C:\projects\rust
git checkout -qf 2564711e803f62e04bebf10408cc1c11297c0caf

[alexcrichton]

@aidanhs thanks for the investigation! Should we file an upstream travis bug for that?

[aidanhs]

@alexcrichton nah I'll just make a PR (resurrecting the one I linked above) at some point in the next week or so. Well, you can raise an issue if you want one for tracking purposes :)

[alexcrichton]

Sounds good to me, thanks!

[aidanhs]

Some updates:

1. Appveyor caching is merged, but the caching is currently consistently not working (the paranoia I've put in place means it's not doing any damage, don't worry!) and I'm trying to investigate using Add a comment for disabling errexit, try to debug appveyor cache #41157. Currently it looks like the created cache zip file is corrupt, so I'm thinking I may need to get in touch with appveyor.
2. I raised an issue on the travis-ci repo about fetch inefficiency, just to provide some context of the issue to the travis people since it turns out there are a few different facets of this problem. Still working on the PR for them.
3. New item: I noticed that the appveyor cache isn't uniquely failing on the git stuff, it can apparently also fail on llvm. This concerns me a bit, since my experience with the git stuff leads me to believe that the failure can result in a partially restored set of files. Since LLVM builds are file based this may not actually be an issue, but it's a bit concerning nonetheless. Currently wondering if it's something to do with synchronisation - it seems that builds for open source projects happen in series, whereas rust is in parallel, so perhaps saving multiple items to the cache at a time causes corruption?

[aidanhs]

Some updates since I've paused work on this (partially to rethink, see 1b):

1. Both Appveyor and travis caching have been removed in Disable git caches again #41398.
   a. Appveyor caching wasn't working anyway due to Add a comment for disabling errexit, try to debug appveyor cache #41157 (comment), which I've not yet got round to ask the appveyor people about. As noted above, I'm currently vaguely suspicious of concurrent cache saves causing corruption but have no evidence.
   b. Travis was timing out and killing cache restores, leaving the git repo in an inconsistent state which wasn't being picked up by the paranoia checks I added. More paranoia is possible, but we probably need to consider a better solution if cache restores are timing out.
2. Following the issue I raised on the travis-ci repo (mentioned in previous comment), I've made a PR on the travis-build repo to correctly do shallow clones on branches, which would make our no-op builds faster.
3. The init_repo.sh script is still in place so intermittent git failures should be correctly handled - the caching I was attempting was just to speed it up, especially on appveyor.

[aidanhs]

Raised http://help.appveyor.com/discussions/problems/6735-corrupt-caches about the corrupt caches.

We've had to implement @notriddle's suggestion to 'temporarily' use .tar.gz files for the llvm submodule in #42211 since it was causing builds to timeout when combined with the current appveyor network issues. I'm not delighted, but while caches don't work there aren't really any other great options I'm aware of.

[aidanhs]

Appveyor have replied to the issue effectively acknowledging the problem and saying it'll be fixed with cache "v2".

[steveklabnik]

Triage: not aware of any movement here

[bjorn3]

We now use Github Actions instead of Travis and AppVeyor. For git submodules we do some magic whereby we download an archive containing just the right version we need and the trick git into thinking it is a regular submodule I believe. Can this issue be closed?