[TEST] Broker / EdgeHub Packaging and Build Infra #3132

and-rewsmith · 2020-06-23T16:59:57Z

The MQTT Broker is designed to run in the same container as EdgeHub Core. Currently, we do not have the packaging logic or build infrastructure to do this.

This PR contains the following:

Packaging logic for edge hub and the mqtt broker to run inside the same container.We made our own watchdog for these two processes (edgehub and broker) implemented in rust.
New build logic for the mqtt broker which will cross compile and lock down necessary dependencies (rust toolchain, openssl, cmake) on a per-platform basis. This is modeled after the edgelet build process defined in package.sh.
Pipeline integration that will build and publish combined edgehub/broker images for each target platform.

Packaging logic
I created a watchdog rust program that will launch two child processes for edgehub and broker. This is the big picture:

Both processes run together in the same container
If either of these processes stop, the other is shut down gracefully and the container exits
If the docker container is stopped, both processes shut down gracefully
Handle processes appropriately (i.e. prevention of zombie processes)
Redirect output of both processes to stdout

Since the watchdog is a component running inside edgehub, I have refactored the directory structure of edgehub to contain subfolders: (docker, core, watchdog). This change bloated the size of the PR as I had to change many .cs and .csproj files.

New Build Logic
We needed to come up with a new build process for the broker and the watchdog. The current approach did not provide flexibility in selecting rust toolchain, openssl, cmake versions.

We modeled this new build logic after edgelet. See edgelet's package.sh for context on how edgelet cross compiles to lock down dependencies. This can mostly be replicated, but since there was no existing logic for alpine, we had to come up with something new.

Regarding alpine, we don't need to build directly on alpine. Building on bionic will also work as long as it produces a musl binary. In order to build musl artifacts on bionic, I ripped out the relevant parts of this dockerfile (line 38-70, 110-126, 145-155), which then get copied into the alpine image.

All the other build logic has precedent in edgelet.

Pipeline Integration
Since we decided not to build windows images for the broker, I have removed windows from the build and manifest steps. I then used the new build logic to build the watchdog/broker artifacts, copied them into the build context (edgehub's dotnet publish folder), then set the watchdog as the image entry point.

This change will significantly increase the build time of our build images pipeline (60+ mins). We can slightly mitigate this by reducing the 6 rust build steps to 3 (consolidate broker and watchdog cross compilations), but the cross compilation of the broker is the most costly. The best way to mitigate this is to have a base image that gets regularly uploaded to our build container registry, but then we need to deal with maintaining compatibility for all feature branches. We can handle this outside this PR.

Thoughts and lingering things

Our current edge hub image build approach is to copy all the contents of the edge hub publish folder into the image. The broker and watchdog artifacts are now inside this publish folder (to reduce build context size). There is currently precedent for this behavior (i.e. the dockerfile folder), however I think this is a dangerous habit because it can lead to undesirable files bloating the container size. This would be acceptable imo if we could provide some sort of exclude list, however that is not supported. Relevant issue. I tried using a rm -rf command to remove the watchdog, mqtt, and docker folders but since the build agent is amd64, it can't run this command on the arm base images. We need to either find a way around this issue or concede to a larger build context containing all the published dotnet apps.
We still have windows docker files checked into the repo. I think it is best to remove them, but was hesitant due to the already large size of the PR.
Need to handle build images release. We will end up replicating the logic in our build pipeline, but how are we going to handle the code sign? I think this might be fine since there is no signing of executables on linux but need to double check.
I am still doing manual testing on the broker, but there is likely some configuration that I could be missing. Let me know if you see something wrong.

and-rewsmith · 2020-06-23T17:01:19Z

edge-hub/docker/linux/arm32v7/Dockerfile

+ADD ./watchdog/arm32/watchdog /usr/local/bin/watchdog
+ADD ./mqtt/arm32/mqttd /usr/local/bin/mqttd


I think we should strip these executables here and in other dockerfiles where appropriate. Unfortunately I doubt it will work here for arm because the build agent is amd64. I think we will need to do it in the build container. It looks like we can use cargo strip to simplify this:
rust-lang/cargo#3483

builds/misc/templates/copy-broker-watchdog-artifacts.yaml

and-rewsmith · 2020-06-23T17:06:06Z

edge-hub/docker/linux/amd64/Dockerfile

+CMD echo "$(date --utc +"%Y-%m-%d %H:%M:%S %:z") Starting Edge Hub" && \
+    /usr/local/bin/watchdog


The broker needs to be configured to dump its state where it has permission to do so.

and-rewsmith · 2020-06-23T17:07:27Z

edge-hub/watchdog/linux/Makefile

+all:
+	$(CARGO) build $(CARGOFLAGS)


I followed the same pattern as for edgelet, but should we remove this? It might be safer to only build release since the only place this makefile is used is in the build pipeline.