Use CircleCI & AppVeyor for continuous & release deployment

[kinke]

No description provided.

[kinke]

Looking very good. Summary:

• LDC-LLVM now has AppVeyor & CircleCI integration; the Windows/Linux/OSX x86_64 and Windows x86 packages (each with enabled and disabled assertions) are uploaded to the GitHub CI release.
• The AppVeyor CI x64/x86 packages for LDC on Windows are now complete, i.e., contain dub, ddemangle, rdmd and dustmite as well as the LICENSE, README.txt and curl.lib files, just like a regular release package.
• CircleCI, now with free OSX support after dropping them a mail, is now also used for a OSX multilib job, something we haven't been able to test so far (their OSX boxes are actually significantly faster than their Linux ones!). 32-bit dmd-testsuite fails due to unexpected warnings for runnable/test13613.d (the output is checked):

ld: warning: could not create compact unwind for __D3std11parallelism16submitAndExecuteFCQBlQBk8TaskPoolMDFZvZv: stack subl instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for __D3std3net4curl4HTTP4Impl15onReceiveHeaderMFNdDFxAaxQdZvZ9__lambda2MFxQvZv: stack subl instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for __D3std8datetime8timezone13PosixTimeZone11getTimeZoneFNeAyaQdZyCQCjQCiQCcQBw: stack subl instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for __D3std9exception11errnoStringFNbNeiZAya: stack subl instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for __D3std5regex8internal9kickstart__T7ShiftOrTaZQl6__ctorMFNcNeKSQCiQChQCe2ir__T5RegexTaZQjAkZSQDmQDlQDiQDc__TQCvTaZQDb: stack subl instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for __D3std7process16spawnProcessImplFNexAAaSQBm5stdio4FileQpQrxHAyaAyaEQCnQCm6ConfigxQBsZCQDgQDf3Pid: stack subl instruction is too different from dwarf stack size

The dynamic-compile lit tests are also failing for the OSX multilib job (ld: library not found for -lldc-jit-rt), the 32/64-bit libs merging step probably isn't performed yet.

• After running the tests successfully, the Linux/OSX CircleCI jobs now also generate a fully-fledged release package and publish it as CircleCI artifact (and upload it to a future GitHub CI release if the commit is on the master branch, just like AppVeyor). In order to generate a multilib package, I enabled MULTILIB for the CircleCI Linux job and moved from the gcc to the ubuntu:14.04 Docker image (+4 minutes for installing dependencies...).

What's left, besides fixing the CircleCI OSX job, is treating tagged builds slightly differently. I'd like to use the LLVM build with enabled assertions for regular CI builds and the one without asserts for tagged builds. The other difference is that the GitHub release to upload the artifact to is a different one and that the filename needs to be adapted.

So after this tedious work,

• most release packages are generated automatically,
• we provide CI builds for the 3 major platforms,
• we have the reassurance that all of these packages passed our testsuite,
• the build and Docker scripts in the ldc-scripts repo are obsoleted,
• and we can easily add some integration tests (vibe.d etc.).

[PetarKirov]

(+4 minutes for installing dependencies...).

You can create a docker image based on ubuntu:14.04, but with all dependencies pre-installed, if you want speed up the Linux CI builds a little more. As a bonus you would have ready image which you simply docker run -it <image>, clone ldc's repo and start building, without having to install or configure anything.
(E.g. everything up to and including Install LDC host compiler can be put in a docker image for Linux.)

[kinke]

I know, we've been using a Docker image for the official release packages for some months now. While it would save about 4-5 mins for each CircleCI Linux job (depending on whether it also includes LLVM, as our existing Docker image does), it would also shift some of the maintenance burden back to us instead of letting the CI services take care of almost everything, which is my primary motivation.

[kinke]

CircleCI Linux now fails to build dub with assertions-enabled LLVM; it's apparently built with optimizations and -g, leading to #2361. No problems on Windows in this regard.

The libFuzzer lit tests fail sporadically and rather frequently for CircleCI OSX Linux, I'd guess about once out of 6 runs, which is annoying. @JohanEngelen: Please look into that if you find the time.

[kinke]

Somebody on OSX please verify that the package works as expected (CircleCI OSX log -> click on Artifacts tab at the top, or just use this link) and confirm that Xcode 9.0.1 in combination with MACOSX_DEPLOYMENT_TARGET=10.8 is fine (9.1 would be available too, as older 8.x versions). Note that there are a few warnings clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9.

I tested the Windows and Linux packages, they are fine. The Windows multilib artifact and GitHub uploads can only be tested after merging this; the release packages will be tested on the next LDC tag.

[dnadlinger]

Seems to work fine on macOS 10.12 after some cursory tests. #2442 applies to all release packages, though.

[dnadlinger]

(Xcode 9.0.1 with a 10.8 deployment target should be fine; this is what we have been using for previous releases.)

[kinke]

Thx for testing, David.

For those who wonder, the LDC-LLVM 5.0.0-2 GitHub artifacts (used by LDC's CircleCI and AppVeyor) have been obtained by downloading, renaming and re-uploading the latest LLVM CI build artifacts (and so the name of the top-level directory in the archives doesn't match the archive name, and the LLVM version string includes the git-... suffix). LLVM 5.0.1 should be released these days, so it should hardly matter.

Wrt. improved OSX test coverage, we not only test MULTILIB=ON now, but also BUILD_SHARED_LIBS=BOTH (e.g., the CircleCI OSX job performs 4x as many druntime/Phobos unittests as a Travis OSX job - and additionally, all of that with enabled LLVM/LDC assertions).

[jondegenhardt]

Just a reminder about the issue found with Ubuntu 14.04: #2390. It's not at all the same thing, but wanted to raise it in case a switch to 16.04 is warranted. (The reason the issue is closed is because 16.04 avoids it.)

[kinke]

Creating a release is incredibly painless now. Only caveat is that the GitHub release should be published (no draft) as soon as the tag is pushed; AppVeyor needs it (stable download links) in order to download the previously generated x64 artifact for generating the multilib one as part of the x86 job.

So the overall release workflow is simplified to:

1. prepare GitHub release draft
2. tag & publish GitHub release
3. wait for all packages to be uploaded (~1 hour), then update ldc-developers.github.io and announce

The armhf package is the only file which still needs to be generated and uploaded manually.