diff --git a/docs/contributing-to-lagoon/developing-lagoon.md b/docs/contributing-to-lagoon/developing-lagoon.md index 9007cafeb3..fbaac9fdb1 100644 --- a/docs/contributing-to-lagoon/developing-lagoon.md +++ b/docs/contributing-to-lagoon/developing-lagoon.md @@ -1,77 +1,182 @@ # Developing Lagoon -Development of Lagoon happens locally via [Docker](https://docs.docker.com/get-docker/). We are using the new [Docker Multi Stage builds](https://docs.docker.com/engine/userguide/eng-image/multistage-build/) very heavily, so it requires at least Docker version 17.06.1. +Development of Lagoon locally can now be performed on a local Kubernetes cluster, or via Docker Compose \(as a fallback\). -## Install Docker and Docker Compose +## Docker -Please check the [official Docs of Docker](https://docs.docker.com/engine/installation/) for how to install Docker. +Docker must be installed to build and run Lagoon locally. -### Docker for Mac +### Install Docker and Docker Compose -Docker Compose is included in Docker for Mac installations. +Please check the [official docs](https://docs.docker.com/engine/installation/) for how to install Docker. -### On Linux - Install Docker Compose +Docker Compose is included in Docker for Mac installations. For Linux installations [see the directions here](https://docs.docker.com/compose/install/). -For Linux installations, [see the directions here](https://docs.docker.com/compose/install/). +### Configure Docker -## Install a Virtual Machine +You will need to update your insecure registries in Docker. [Read the instructions here on how to do that](https://docs.docker.com/registry/insecure/). We suggest adding the entire local IPv4 Private Address Spaces to avoid unnecessary reconfiguration between Kubernetes and Docker Compose. e.g. `"insecure-registries" : ["172.16.0.0/12","192.168.0.0/16"],` -### On GNU/Linux hosts +### Allocate Enough Docker Resources -For GNU/Linux hosts, we are using KVM \(Kernel-based Virtual Machine\) as a default virtualization engine to run Openshift Minishift VM. [Read the installation instructions here](https://docs.okd.io/latest/minishift/getting-started/setting-up-virtualization-environment.html#for-linux). +Running a Lagoon, Kubernetes, or Docker cluster on your local machine consumes a lot of resources. We recommend that you give your Docker host a minimum of 8 CPU cores and 12GB RAM. -### Install VirtualBox on other hosts +## Build Lagoon Locally -For hosts other than GNU/Linux, we are using VirtualBox to run the Openshift Minishift VM. [For download and installation instructions see here](https://www.virtualbox.org/). +{% hint style="warning" %} +Only consider building Lagoon this way if you intend to develop features or functionality for it, or want to debug internal processes. We will also be providing instruction to install Lagoon without building it \(i.e. by using the published releases\). -## Start Services +We're using `make` \(see the [Makefile](https://github.com/uselagoon/lagoon/blob/main/Makefile)\) in order to build the needed Docker images, configure Kubernetes and run tests. +{% endhint %} -1. Add `192.168.42.0/24` to insecure registries in Docker. [Read the instructions here on how to do that](https://docs.docker.com/registry/insecure/). -2. Also make sure that you give your Docker host a minimum of 4 CPUs and 4GB Ram. +We have provided a number of routines in the [Makefile](https://github.com/uselagoon/lagoon/blob/main/Makefile) to cover most local development scenarios. Here we will run through a complete process. -{% hint style="warning" %} -Lagoon consists of a lot of services and Docker images. Building and running them locally might not even be necessary. +### Build images + +1. Here `-j8` tells **make** to run 8 tasks in parallel to speed the build up. Adjust as necessary. +2. We have set `SCAN_IAMGES=false` as a default to not scan the built images for vulnerabilities. If set to true, a `scan.txt` file will be created in the project root with the scan output. + +```bash +make -j8 build +``` + +3. Start Lagoon test routine using the defaults in the Makefile \(all tests\). + +```bash +make kind/test +``` -We're using `make` \(see the [Makefile](https://github.com/uselagoon/lagoon/blob/main/Makefile)\) in order to only build the needed Docker images specifically for a part of Lagoon. +{% hint style="warning" %} +There are a lot of tests configured to run by default - please consider only testing locally the minimum that you need to ensure functionality. This can be done by specifying or removing tests from the `TESTS` variable in the Makefile. {% endhint %} -All of this is based around tests. So if you want to only build the part that is needed to work on the Node.js deployment, for example, you can run the tests with `make tests/node`, and this will then set up all the needed stuff for the Node.js deployment part \(OpenShift, building images, services\). +This process will: -If you still want to build and start all services, go ahead: +1. Download the correct versions of the local development tools if not installed - `kind`, `kubectl`, `helm`, `jq`. +2. Update the necessary Helm repositories for Lagoon to function. +3. Ensure all of the correct images have been built in the previous step. +4. Create a local [KinD](https://kind.sigs.k8s.io/) cluster, which provisions an entire running Kubernetes cluster in a local Docker container. This cluster has been configured to talk to a provisioned image registry that we will be pushing the built Lagoon images to. It has also been configured to allow access to the host filesystem for local development. +5. Clone Lagoon from [https://github.com/uselagoon/lagoon-charts](https://github.com/uselagoon/lagoon-charts) \(use the `CHARTS_TREEISH` variable in the Makefile to control which branch if needed\). +6. Install the Harbor Image registry into the KinD cluster and configure its ingress and access properly. +7. Docker will push the built images for Lagoon into the Harbor image registry. +8. It then uses the [Makefile from lagoon-charts](https://github.com/uselagoon/lagoon-charts/blob/main/Makefile) to perform the rest of the setup steps. +9. A suitable ingress controller is installed - we use the [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/). +10. A local NFS server provisioner is installed to handle specific volume requests - we use one that handles Read-Write-Many operations \(RWX\). +11. Lagoon Core is then installed, using the locally built images pushed to the cluster-local Image Registry, and using the default configuration, which may exclude some services not needed for local testing. The installation will wait for the API and Keycloak to come online. +12. The DBaaS providers are installed - MariaDB, PostgreSQL and MongoDB. This step provisions standalone databases to be used by projects running locally, and emulates the managed services available via cloud providers \(e.g. Cloud SQL, RDS or Azure Database\). +13. Lagoon Remote is then installed, and configured to talk to the Lagoon Core, databases and local storage. The installation will wait for this to complete before continuing. +14. To provision the tests, the Lagoon Test chart is then installed, which provisions a local Git server to host the test repositories, and pre-configures the Lagoon API database with the default test users, accounts and configuration. It then performs readiness checks before starting tests. +15. Lagoon will run all the tests specified in the TESTS variable in the Makefile. Each test creates its own project & environments, performs the tests, and then removes the environments & projects. The test runs are output to the console log in the `lagoon-test-suite-*` pod, and can be accessed one test per container. -1. Build images: +Ideally, all of the tests pass and it's all done! + +### View the test progress and your local cluster + +The test routine creates a local Kubeconfig file \(called `kubeconfig.kind.lagoon` in the root of the project, that can be used with a Kubernetes dashboard, viewer or CLI tool to access the local cluster. We use tools like [Lens](https://k8slens.dev/), [Octant](https://octant.dev/), [kubectl](https://kubernetes.io/docs/reference/kubectl/cheatsheet/) or [Portainer](https://www.portainer.io/) in our workflows. Lagoon Core, Remote and Tests all build in the `Lagoon` namespace, and each environment creates its own namespace to run, so make sure to use the correct context when inspecting. + +In order to use kubectl with the local cluster, you will need to use the correct Kubeconfig. This can be done for every command or it can be added to your preferred tool: ```bash -make build +KUBECONFIG=./kubeconfig.kind.lagoon kubectl get pods -n lagoon ``` -1. Start Lagoon services: +The Helm charts used to build the local Lagoon are cloned into a local folder and symlinked to `lagoon-charts.kind.lagoon` where you can see the configuration. We'll cover how to make easy modifications later in this documentation. + +### Interact with your local Lagoon cluster + +The Makefile includes a few simple routines that will make interacting with the installed Lagoon simpler: ```bash -make up +make kind/port-forwards ``` -1. Follow the services logs: +This will create local ports to expose the UI \(6060\), API \(7070\) and Keycloak \(8080\). Note that this logs to `stdout`, so it should be performed in a secondary terminal/window. ```bash -make logs +make kind/get-admin-creds ``` -1. Run tests \(read [Tests](tests.md) to learn more about testing\): +This will retrieve the necessary credentials to interact with the Lagoon. + +* The JWT is an admin-scoped token for use as a bearer token with your local GraphQL client. [See more in our GraphQL documentation](../using-lagoon-advanced/graphql.md). +* There is a token for use with the "admin" user in Keycloak, who can access all users, groups, roles, etc. +* There is also a token for use with the "lagoonadmin" user in Lagoon, which can be allocated default groups, permissions, etc. ```bash -make tests +make kind/dev ``` -1. Check out what happens in OpenShift \(credentials: `developer`/`developer`\): +This will re-push the images listed in `KIND_SERVICES` with the correct tag, and redeploy the lagoon-core chart. This is useful for testing small changes to Lagoon services, but does not support "live" development. You will need to rebuild these images locally first, e.g `rm build/api && make build/api`. ```bash -echo "visit https://$(minishift --profile lagoon ip):8443/console" +make kind/local-dev-patch ``` +This will build the typescript services, using your locally installed Node.js \(it should be >16.0\). It will then: + +* Mount the "dist" folders from the Lagoon services into the correct lagoon-core pods in Kubernetes +* Redeploy the lagoon-core chart with the services running with `nodemon`watching the code for changes +* This will facilitate "live" development on Lagoon. +* Note that occasionally the pod in Kubernetes may require redeployment for a change to show. Clean any build artifacts from those services if you're rebuilding different branches with `git clean -dfx` as the dist folders are ignored by Git. + +```bash +make kind/local-dev-logging +``` + +This will create a standalone OpenDistro for Elasticsearch cluster in your local Docker, and configure Lagoon to dispatch all logs \(Lagoon and project\) to it, using the configuration in [lagoon-logging](https://github.com/uselagoon/lagoon-charts/tree/main/charts/lagoon-logging). + +```bash +make kind/retest +# OR +make kind/retest TESTS='[features-kubernetes]' +``` + +This will re-run a suite of tests \(defined in the `TESTS` variable\) against the existing cluster. It will re-push the images needed for tests \(tests, local-git, and the data-watcher-pusher\). You can specify tests to run by passing the TESTS variable inline. + +If updating a test configuration, the tests image will need to be rebuilt and pushed, e.g `rm build/tests && make build/tests && make kind/push-images IMAGES='tests' && make kind/retest TESTS='[api]'` + +```bash +make kind/push-images +# OR +make kind/push-images IMAGES='tests local-git' +``` + +This will push all the images up to the image registry. Specifying `IMAGES` will tag and push specific images. + +```bash +make kind/clean +``` + +This will remove the KinD Lagoon cluster from your local Docker. + +### Ansible + +The Lagoon test uses Ansible to run the test suite. Each range of tests for a specific function has been split into its own routine. If you are performing development work locally, select which tests to run, and update the `$TESTS` variable in the Makefile to reduce the concurrent tests running. + +The configuration for these tests is held in three services: + +* `tests` is the Ansible test services themselves. The local testing routine runs each individual test as a separate container within a test-suite pod. These are listed below. +* `local-git` is a Git server hosted in the cluster that holds the source files for the tests. Ansible pulls and pushes to this repository throughout the tests +* `api-data-watcher-pusher` is a set of GraphQL mutations that pre-populates local Lagoon with the necessary Kubernetes configuration, test user accounts and SSH keys, and the necessary groups and notifications. **Note that this will wipe local projects and environments on each run.** + +The individual routines relevant to Kubernetes are: + +* `active-standby-kubernetes` runs tests to check active/standby in Kubernetes. +* `api` runs tests for the API - branch/PR deployment, promotion. +* `bitbucket`, `gitlab` and `github` run tests for the specific SCM providers. +* `drupal-php74` runs a single-pod MariaDB, MariaDB DBaaS and a Drush-specific test for a Drupal 8/9 project \(`drupal-php73` doesn't do the Drush test\). +* `drupal-postgres` runs a single-pod PostgreSQL and a PostgreSQL DBaaS test for a Drupal 8 project. +* `elasticsearch` runs a simple NGINX proxy to an Elasticsearch single-pod. +* `features-api-variables` runs tests that utilize variables in Lagoon. +* `features-kubernetes` runs a range of standard Lagoon tests, specific to Kubernetes. +* `features-kubernetes-2` runs more advanced kubernetes-specific tests - covering multi-project and subfolder configurations. +* `nginx`, `node` and `python` run basic tests against those project types. +* `node-mongodb` runs a single-pod MongoDB test and a MongoDB DBaaS test against a Node.js app. + +There are a few other legacy Openshift-specific tests in there that may or may not work with Openshift-based clients. + ## Local Development -Most services are written in [Node.js](https://nodejs.org/en/docs/). As many of these services share similar Node.js code and Node.js packages, we're using a new feature of [Yarn](https://yarnpkg.com/en/docs), called [Yarn workspaces](https://yarnpkg.com/en/docs/workspaces). Yarn workspaces need a `package.json` in the project's root directory that defines the workspaces. +Most services are written in [Node.js](https://nodejs.org/en/docs/). As many of these services share similar Node.js code and Node.js packages, we're using a feature of [Yarn](https://yarnpkg.com/en/docs), called [Yarn workspaces](https://yarnpkg.com/en/docs/workspaces). Yarn workspaces need a `package.json` in the project's root directory that defines the workspaces. The development of the services can happen directly within Docker. Each container for each service is set up in a way that its source code is mounted into the running container \([see `docker-compose.yml`](../using-lagoon-the-basics/docker-compose-yml.md)\). Node.js itself is watching the code via `nodemon` , and restarts the Node.js process automatically on a change. @@ -79,33 +184,110 @@ The development of the services can happen directly within Docker. Each containe The services not only share many Node.js packages, but also share actual custom code. This code is within `node-packages/lagoon-commons`. It will be automatically symlinked by Yarn workspaces. Additionally, the [`nodemon`](https://www.npmjs.com/package/nodemon) of the services is set up in a way that it checks for changes in `node-packages` and will restart the node process automatically. -### Hiera - -The API uses a [Puppet](https://puppet.com/docs/puppet/latest/puppet_index.html)-compatible YAML format called [Hiera](https://puppet.com/docs/puppet/latest/hiera.html) to store its data. On production, this Hiera is in another Git repository. For local development, there is a folder called `local-hiera` which contains test data that is used during development and testing, plus it has no client related data. For easier development, there is `local-hiera-watcher-pusher`, which watches the `local-hiera` folder. On every change, it pushes the changes into `local-git-server`, which emulates a Git server just like it is on production. The API service is connecting to this local Git server and updates its data from the server. - ## Troubleshooting -⚠ **I can't build a docker image for any Node.js based service** +#### ⚠ **I can't build a docker image for any Node.js based service** -Rebuild the images via +Rebuild the images via: ```bash make clean make build ``` -⚠ **I get errors about missing node\_modules content when I try to build / run a Node.js based image** +#### ⚠ **I get errors about missing `node_modules` content when I try to build / run a Node.js based image** Make sure to run `yarn` in Lagoon's root directory, since some services have common dependencies managed by `yarn` workspaces. -⚠ **My builds can't resolve domains** +#### ⚠ **I get an error resolving the `nip.io` domains** + +```text +Error response from daemon: Get https://registry.172.18.0.2.nip.io:32080/v2/: dial tcp: lookup registry.172.18.0.2.nip.io: no such host +``` + +This can happen if your local resolver filters private IPs from results. You can work around this by editing `/etc/resolv.conf` and adding a line like `nameserver 8.8.8.8` at the top to use a public resolver that doesn't filter results. + +## Example workflows -Some Internet Service Providers \(ISPs\) set up a "search domain" to catch domain name errors. VirtualBox will copy this setting into MiniShift, which can cause domain resolution errors in the OpenShift pods. To check for this problem, look at the `/etc/resolv.conf` in your failing pod and check for errant search domains. +Here are some development scenarios and useful workflows for getting things done. -To fix, you must remove the extra search domain. +### Editing `kubectl-build-deploy-dind` -* Log in to the MiniShift vm: `minishift ssh`. -* Remove the setting from `/etc/resolv.conf`. -* Restart openshift docker: `sudo docker restart origin`. -* Redeploy `docker-host` in the `lagoon` project. +This example shows a workflow for editing the Lagoon deploy logic. + +#### Edit `kubectl-build-deploy-dind` + +In this example we want to add some functionality to the Lagoon deploy logic in the `kubectl-build-deploy-dind` image. + +1. Start a local KinD cluster with Lagoon installed from locally built images, and smoke-test it by running a single test suite: + +```bash +make -j8 kind/test TESTS='[features-api-variables]' +``` + +2. Edit `images/kubectl-build-deploy-dind/build-deploy-docker-compose.sh`. + +```diff +--- a/images/kubectl-build-deploy-dind/build-deploy-docker-compose.sh ++++ b/images/kubectl-build-deploy-dind/build-deploy-docker-compose.sh +@@ -1,5 +1,7 @@ + #!/bin/bash + ++echo HELLO WORLD ++ + function cronScheduleMoreOftenThan30Minutes() { + #takes a unexpanded cron schedule, returns 0 if it's more often that 30 minutes + MINUTE=$(echo $1 | (read -a ARRAY; echo ${ARRAY[0]}) ) +``` + +3. Now rebuild the `kubectl-build-deploy-dind` image with the edits included. + +```bash +rm build/kubectl-build-deploy-dind +make -j8 build/kubectl-build-deploy-dind +``` + +4. Push the newly built image into the cluster registry. It will now be used for future deploys. + +```bash +make kind/push-images IMAGES=kubectl-build-deploy-dind +``` + +5. Rerun the tests. + +```bash +make kind/retest TESTS='[features-api-variables]' +``` + +6. See the edits have been applied. + +```bash +$ kubectl -n ci-features-api-variables-control-k8s-lagoon-api-variables logs lagoon-build-lat2b | grep -A2 build-deploy-docker-compose.sh ++ . /kubectl-build-deploy/build-deploy-docker-compose.sh +++ echo HELLO WORLD +HELLO WORLD +``` + +#### Add tests + +1. Repeat the first step above. +2. Edit `tests/tests/features-api-variables.yaml` and add a test case. +3. Rebuild the `tests` image. + +```bash +rm build/tests +make -j8 build/tests +``` + +4. Push the new `tests` image into the cluster registry. + +```bash +make kind/push-images IMAGES=tests +``` + +5. Rerun the tests. + +```bash +make kind/retest TESTS='[features-api-variables]' +```