🎈 Welcome to the Conduit development guide! 👋
This document will help you build and run Conduit from source. More information about testing from source can be found in the TEST.md guide.
Conduit is primarily written in Rust, Go, and React. At its core is a high-performance data plane written in Rust. The control plane components are written in Go. The dashboard UI is a React application.
cli
: Command-lineconduit
utility, view and drive the control plane.controller
destination
: Serves service discovery information to theproxy
.proxy-api
: Accepts requests fromproxy
instances and forwards those requests to the appropriate controller service.public-api
: Accepts requests from API clients such ascli
andweb
, provides access to and control of the conduit service mesh.tap
: Provides a live pipeline of requests.telemetry
: Collects and aggregates metrics fromproxy
componenets.
proxy-init
: Adds a Kubernetes pod to join the Conduit Service Mesh.web
: Provides a UI dashboard to view and drive the control plane. This component is written in Go and React.
proxy
: High-performance data plane, injected as a sidecar with every service.
conduit_components digraph G { rankdir=LR;
node [style=filled, shape=rect];
"cli" [color=lightblue];
"destination" [color=lightblue];
"proxy-api" [color=lightblue];
"public-api" [color=lightblue];
"tap" [color=lightblue];
"telemetry" [color=lightblue];
"web" [color=lightblue];
"proxy" [color=orange];
"cli" -> "public-api";
"web" -> "public-api";
"destination" -> "kubernetes";
"proxy" -> "proxy-api";
"proxy-api" -> "destination";
"proxy-api" -> "telemetry";
"public-api" -> "tap";
"public-api" -> "telemetry";
"tap" -> "kubernetes";
"tap" -> "proxy";
"telemetry" -> "kubernetes";
"telemetry" -> "prometheus";
} conduit_components
Depending on use case, there are several configurations with which to develop and run Conduit:
- Comprehensive: Integrated configuration using Minikube, most closely matches release.
- Go: Development of the Go components using Docker Compose.
- Web: Development of the Conduit Dashboard.
- Rust: Standalone development of the Rust
proxy
.
This configuration builds all Conduit components in Docker images, and deploys them onto Minikube. This setup most closely parallels our recommended production installation, documented at https://conduit.io/getting-started/.
These commands assume a working Minikube environment.
# build all docker images, using minikube as our docker repo
DOCKER_TRACE=1 bin/mkube bin/docker-build
# install conduit
bin/conduit install | kubectl apply -f -
# verify cli and server versions
bin/conduit version
# validate installation
kubectl --namespace=conduit get all
bin/conduit check --expected-version $(bin/root-tag)
# view conduit dashboard
bin/conduit dashboard
# install the demo app
curl https://raw.githubusercontent.com/runconduit/conduit-examples/master/emojivoto/emojivoto.yml | bin/conduit inject - | kubectl apply -f -
# view demo app
minikube -n emojivoto service web-svc --url
# view details per deployment
bin/conduit stat deployments
# view a live pipeline of requests
bin/conduit tap deploy emojivoto/voting
These commands assume working Go and Docker environments.
To run all of the Go apps in a docker-compose environment:
docker-compose build
docker-compose up -d
# view dashboard
open http://$DOCKER_IP:8084
If your system is configured to talk to a Kubernetes cluster, you can simulate traffic to the docker-compose environment:
# confirm you are connected to Kubernetes
kubectl version
Note that the Kubernetes cluster your system is configured to talk to must not
be referenced via localhost
in your Kubernetes config file, as
simulate-proxy
will not be able to connect to it. This includes Kubernetes on
Docker For Mac.
Our instructions use a bin/go-run
script in lieu go run
.
This is a convenience script that leverages caching via go build
to make your
build/run/debug loop faster.
In general, replace commands like this:
go run cli/main.go
with this:
bin/go-run cli
You may also leverage go-run
to execute our conduit
cli command. While in a
release context you may run:
conduit check
In development you can run:
bin/go-run cli check
Conduit's control plane is composed of several Go microservices. You can run these components in a Kubernetes (or Minikube) cluster, or even locally.
To run an individual component locally, you can use the go-run
command, and
pass in valid Kubernetes credentials via the -kubeconfig
flag. For instance,
to run the destination service locally, run:
bin/go-run controller/cmd/destination -kubeconfig ~/.kube/config -log-level debug
You can send test requests to the destination service using the
destination-client
in the controller/script
directory. For instance:
bin/go-run controller/script/destination-client -path hello.default.svc.cluster.local:80
This is a React app fronting a Go process. It uses webpack to bundle assets, and postcss to transform css.
These commands assume working Go and Yarn environments.
Install Yarn and use it to install dependencies:
brew install yarn
cd web/app
yarn
cd web/app
yarn && yarn webpack
cd ..
../bin/go-run .
The web server will be running on localhost:8084
.
Note the web
process depends on a public-api
server, for which you have
three options:
bin/go-run controller/cmd/public-api
Stop the web service, then run it locally and set the --api-addr
flag to the
address of the public API server that's running in your docker environment:
docker-compose stop web
cd web
../bin/go-run . --api-addr=$DOCKER_IP:8085
If you are running the public API server in Kubernetes, forward localhost:8085
to the Conduit controller pod:
kubectl --namespace=conduit port-forward $(
kubectl --namespace=conduit get po --selector=conduit.io/control-plane-component=controller -o jsonpath='{.items[*].metadata.name}'
) 8085:8085
Then connect the local web process to the forwarded port:
cd web
../bin/go-run . --api-addr=localhost:8085
To develop with a webpack dev server, start the server in a separate window:
cd web/app
yarn webpack-dev-server
And then set the --webpack-dev-server
flag when running the web server:
cd web
../bin/go-run . --webpack-dev-server=http://localhost:8080
To add a JS dependency:
cd web/app
yarn add [dep]
These commands assume a working Rust environment.
Note that we only support the most recent stable
version of Rust.
To build and run the Rust proxy:
cargo build -p conduit-proxy
CONDUIT_PROXY_LOG=trace \
CONDUIT_PROXY_PUBLIC_LISTENER=tcp://0.0.0.0:5432 \
CONDUIT_PROXY_PRIVATE_FORWARD=tcp://127.0.0.1:1234 \
CONDUIT_PROXY_CONTROL_URL=tcp://127.0.0.1:8086 \
target/debug/conduit-proxy
To connect to a live proxy-api
at localhost:8086
:
bin/go-run controller/cmd/proxy-api
The bin/docker-build-proxy
script builds the proxy:
DOCKER_TRACE=1 PROXY_UNOPTIMIZED=1 PROXY_SKIP_TESTS=1 bin/docker-build-proxy
It supports two environment variables:
PROXY_UNOPTIMIZED
-- When set and non-empty, produces unoptimized build artifacts, which reduces build times at the expense of runtime performance. Changing this will likely invalidate a substantial portion of Docker's cache.PROXY_SKIP_TESTS
-- When set and non-empty, prevents the proxy's tests from being run during the build. Changing this setting will not invalidate Docker's cache.
If you make Protobuf changes, run:
bin/dep ensure
bin/protoc-go.sh
The Rust proxy and Go Docker images rely on base dependency images with hard-coded SHA's:
gcr.io/runconduit/go-deps
depends on
bin/update-go-deps-shas
must be run when go dependencies change.
build_architecture digraph G { rankdir=LR;
"Dockerfile-base" [color=lightblue, style=filled, shape=rect];
"Dockerfile-go-deps" [color=lightblue, style=filled, shape=rect];
"controller/Dockerfile" [color=lightblue, style=filled, shape=rect];
"cli/Dockerfile-bin" [color=lightblue, style=filled, shape=rect];
"grafana/Dockerfile" [color=lightblue, style=filled, shape=rect];
"proxy/Dockerfile" [color=lightblue, style=filled, shape=rect];
"proxy-init/Dockerfile" [color=lightblue, style=filled, shape=rect];
"proxy-init/integration_test/iptables/Dockerfile-tester" [color=lightblue, style=filled, shape=rect];
"web/Dockerfile" [color=lightblue, style=filled, shape=rect];
"proxy-init/integration_test/run_tests.sh" -> "proxy-init/integration_test/iptables/Dockerfile-tester";
"_docker.sh" -> "_log.sh";
"_gcp.sh";
"_log.sh";
"_tag.sh";
"conduit" -> "docker-build-cli-bin";
"dep";
"docker-build" -> "docker-build-cli-bin";
"docker-build" -> "docker-build-controller";
"docker-build" -> "docker-build-grafana";
"docker-build" -> "docker-build-proxy";
"docker-build" -> "docker-build-proxy-init";
"docker-build" -> "docker-build-web";
"docker-build-base" -> "_docker.sh";
"docker-build-base" -> "Dockerfile-base";
"docker-build-cli-bin" -> "_docker.sh";
"docker-build-cli-bin" -> "_tag.sh";
"docker-build-cli-bin" -> "docker-build-base";
"docker-build-cli-bin" -> "docker-build-go-deps";
"docker-build-cli-bin" -> "cli/Dockerfile-bin";
"docker-build-controller" -> "_docker.sh";
"docker-build-controller" -> "_tag.sh";
"docker-build-controller" -> "docker-build-base";
"docker-build-controller" -> "docker-build-go-deps";
"docker-build-controller" -> "controller/Dockerfile";
"docker-build-go-deps" -> "_docker.sh";
"docker-build-go-deps" -> "_tag.sh";
"docker-build-go-deps" -> "Dockerfile-go-deps";
"docker-build-grafana" -> "_docker.sh";
"docker-build-grafana" -> "_tag.sh";
"docker-build-grafana" -> "grafana/Dockerfile";
"docker-build-proxy" -> "_docker.sh";
"docker-build-proxy" -> "_tag.sh";
"docker-build-proxy" -> "proxy/Dockerfile";
"docker-build-proxy-init" -> "_docker.sh";
"docker-build-proxy-init" -> "_tag.sh";
"docker-build-proxy-init" -> "docker-build-base";
"docker-build-proxy-init" -> "docker-build-go-deps";
"docker-build-proxy-init" -> "proxy-init/Dockerfile";
"docker-build-web" -> "_docker.sh";
"docker-build-web" -> "_tag.sh";
"docker-build-web" -> "docker-build-base";
"docker-build-web" -> "docker-build-go-deps";
"docker-build-web" -> "web/Dockerfile";
"docker-images" -> "_docker.sh";
"docker-images" -> "_tag.sh";
"docker-pull" -> "_docker.sh";
"docker-pull-deps" -> "_docker.sh";
"docker-pull-deps" -> "_tag.sh";
"docker-push" -> "_docker.sh";
"docker-push-deps" -> "_docker.sh";
"docker-push-deps" -> "_tag.sh";
"docker-retag-all" -> "_docker.sh";
"go-run" -> ".gorun";
"go-run" -> "root-tag";
"minikube-start-hyperv.bat";
"mkube";
"protoc" -> ".protoc";
"protoc-go.sh" -> "protoc";
"root-tag" -> "_tag.sh";
"test-cleanup";
"test-run";
".travis.yml" -> "_gcp.sh";
".travis.yml" -> "_tag.sh";
".travis.yml" -> "dep";
".travis.yml" -> "docker-build";
".travis.yml" -> "docker-pull";
".travis.yml" -> "docker-pull-deps";
".travis.yml" -> "docker-push";
".travis.yml" -> "docker-push-deps";
".travis.yml" -> "docker-retag-all";
".travis.yml" -> "protoc-go.sh";
"update-go-deps-shas" -> "_tag.sh";
"update-go-deps-shas" -> "cli/Dockerfile-bin";
"update-go-deps-shas" -> "controller/Dockerfile";
"update-go-deps-shas" -> "proxy-init/Dockerfile";
"update-go-deps-shas" -> "web/Dockerfile";
} build_architecture