From f3fd9e6432db68ecab3c1a0219a8300f6c181d3d Mon Sep 17 00:00:00 2001 From: Ryan Lovett Date: Wed, 20 Nov 2024 15:35:09 -0800 Subject: [PATCH 1/3] Add local repo2docker information. Add a new file for running repo2docker locally. Also shorten a few headings in the new image doc. --- docs/_quarto.yml | 1 + docs/tasks/new-image.qmd | 93 ++++++++++++++++---------------- docs/tasks/repo2docker-local.qmd | 64 ++++++++++++++++++++++ 3 files changed, 111 insertions(+), 47 deletions(-) create mode 100644 docs/tasks/repo2docker-local.qmd diff --git a/docs/_quarto.yml b/docs/_quarto.yml index f7b946b1d..4340e38ed 100644 --- a/docs/_quarto.yml +++ b/docs/_quarto.yml @@ -46,6 +46,7 @@ website: - tasks/rebuild-postgres-image.qmd - tasks/managing-multiple-user-image-repos.qmd - tasks/new-image.qmd + - tasks/repo2docker-local.qmd - tasks/transition-image.qmd - tasks/new-packages.qmd - tasks/course-config.qmd diff --git a/docs/tasks/new-image.qmd b/docs/tasks/new-image.qmd index c15fc0d74..e6b87aa6d 100644 --- a/docs/tasks/new-image.qmd +++ b/docs/tasks/new-image.qmd @@ -19,22 +19,20 @@ If that is the case, only a `Dockerfile` format will work. As always, create a feature branch for your changes, and submit a PR when done. -## Use an existing image as a template +There are two approaches to pre-populate the image's assets: -Browse through our [image repos](https://github.com/orgs/berkeley-dsep-infra/repositories?language=&q=image&sort=&type=all) -to find a hub that is similar to the one you are trying to create. This will -give you a good starting point. + - Use an existing image as a template. + Browse through our [image +repos](https://github.com/orgs/berkeley-dsep-infra/repositories?language=&q=image&sort=&type=all) + to find a hub that is similar to the one you are trying to create. This will + give you a good starting point. -## Create the image repos + - Fork [hub-user-image-template](https://github.com/berkeley-dsep-infra/hub-user-image-template). Click "Use this template" > "Create a new repository". + Be sure to follow convention and name the repo `-user-image`, and + the owner needs to be `berkeley-dsep-infra`. When that is done, create your + own fork of the new repo. -Create a new image repo from the [hub-user-image-template](https://github.com/berkeley-dsep-infra/hub-user-image-template). -Click "Use this template" > "Create a new repository". - -Be sure to follow convention and name the repo `-user-image`, and the -owner needs to be `berkeley-dsep-infra`. When that is done, create your own -fork of the new repo. - -### Configuring the root image repo +### Image Repository Settings There are now a few steps to set up the CI/CD for the new image repo. In the `berkeley-dsep-infra` image repo, click on `Settings`, and under `General`, @@ -53,7 +51,7 @@ will be adding two new variables: always be `ucb-datahub-2018/user-images/` and the image name will always be the same as the repo: `-user-image`. -### Configure your fork +### Your Fork's Repository Settings Now you will want to disable Github Actions for your fork of the image repo. If you don't, whenever you push PRs to the root repo the workflows *in your @@ -64,25 +62,29 @@ failure. To disable this for your fork, click on `Settings`, `Actions` and `General`. Check the `Disable actions` box and click save. -### Add the root image repo to the list of allowed repos in the `berkeley-dsep-infra` secrets. +### Enable Artifact Registry Pushing -Now, go to the `berkeley-dsep-infra` [Secrets and Variables](https://github.com/organizations/berkeley-dsep-infra/settings/secrets/actions). -You will need to give your repo permissions to push to the Artifact Registry, +The image repository needs to be added to the list of allowed repositories in +the `berkeley-dsep-infra` secrets. Go to the `berkeley-dsep-infra` [Secrets and +Variables](https://github.com/organizations/berkeley-dsep-infra/settings/secrets/actions). +Give your repository permissions to push to the Artifact Registry, as well as to push a branch to the [datahub repo](https://github.com/berkeley-dsep-infra/datahub). Edit both `DATAHUB_CREATE_PR` and `GAR_SECRET_KEY`, and click on the gear icon, search for your repo name, check the box and save. -### Update your deployment's `hubploy.yaml` and add the image to the primary list of repos. +### Configure `hubploy` -You need to let `hubploy` know the specifics of the image. Change the `name` of the image in -`deployments//hubploy.yaml` to point to your new image name, and after the name add -`:PLACEHOLDER` in place of the image sha. This will be automatically updated after your new image -is built and pushed to the Artifact Registry. +You need to let `hubploy` know the specifics of the image by updating your +deployment's `hubploy.yaml`. Change the `name` of the image in +`deployments//hubploy.yaml` to point to your new image name, and after +the name add `:PLACEHOLDER` in place of the image sha. This will be +automatically updated after your new image is built and pushed to the Artifact +Registry. Example: -``` +```yaml images: images: - name: us-central1-docker.pkg.dev/ucb-datahub-2018/user-images/fancynewhub-user-image:PLACEHOLDER @@ -102,7 +104,7 @@ Create a PR and merge to staging. You can cancel the [`Deploy staging and prod hubs` job in Actions](https://github.com/berkeley-dsep-infra/datahub/actions/workflows/deploy-hubs.yaml), or just let it fail. -## Add a github bot notification in Slack +## Add a GitHub Slack Bot Go to the #ucb-datahubs-bots channel, and run the following command: @@ -110,48 +112,45 @@ Go to the #ucb-datahubs-bots channel, and run the following command: /github subscribe berkeley-dsep-infra/ ``` -## Modify the image configuration as necessary +## Modify the Image -This step is straightforward: create a feature branch, edit/modify/delete/add -any files in the image repo to configure the image as needed. +This step is straightforward: create a feature branch, and edit, delete, or add +any files to configure the image as needed. We also strongly recommend copying `README-template.md` over the default `README.md`, and modifying it to replace all occurrences of `` with the name of your image. -## Submitting a pull request +## Submit Pull Requests Familiarize yourself with [pull requests](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests) -and [repo2docker](https://github.com/jupyter/repo2docker) , and create a -fork of the [datahub staging -branch](https://github.com/berkeley-dsep-infra/datahub). - -1. Set up your git/dev environment by [following the instructions - here](https://github.com/berkeley-dsep-infra/hub-user-image-template/blob/main/CONTRIBUTING.md). - : - This guide is also located in your image repo! - -2. Test the changes locally using `repo2docker`, then submit a PR to `staging`. +and [repo2docker](https://github.com/jupyter/repo2docker), and create a fork of +the [datahub staging branch](https://github.com/berkeley-dsep-infra/datahub). - : - To use `repo2docker`, be sure that you are inside the image - repo directory on your device, and then run `repo2docker .`. +1. Set up your git/dev environment by following the [image templat's +contributing + guide](https://github.com/berkeley-dsep-infra/hub-user-image-template/blob/main/CONTRIBUTING.md). -3. Commit and push your changes to your fork of the image repo, and +1. [Test the image locally](repo2docker-local.qmd) using `repo2docker`. +1. Submit a PR to `staging`. +1. Commit and push your changes to your fork of the image repo, and create a new pull request at https://github.com/berkeley-dsep-infra/. -4. After the build passes, merge your PR in to `main` and the image will +1. After the build passes, merge your PR in to `main` and the image will be built again and pushed to the Artifact Registry. If that succeeds, then a commit will be crafted that will update the `PLACEHOLDER` field in `hubploy.yaml` with the image's SHA and pushed to the datahub repo. You can check on the progress of this workflow in your root image repo's `Actions` tab. -5. After 4 is completed successfully, go to the Datahub repo and click on - the [New pull request](https://github.com/berkeley-dsep-infra/datahub/compare) +1. After the previous step is completed successfully, go to the Datahub repo + and click on the [New pull + request](https://github.com/berkeley-dsep-infra/datahub/compare) button. Next, click on the `compare: staging` drop down, and you should see - a branch named something like `update--image-tag-`. Select that, - and create a new pull request. + a branch named something like `update--image-tag-`. Select + that, and create a new pull request. -6. Once the checks has passed, merge to `staging` and your new image will be - deployed! You can watch the progress [here](https://github.com/berkeley-dsep-infra/datahub/actions/workflows/deploy-hubs.yaml). +1. Once the checks has passed, merge to `staging` and your new image will be + deployed! You can watch the progress in the [deploy-hubs workflow](https://github.com/berkeley-dsep-infra/datahub/actions/workflows/deploy-hubs.yaml). diff --git a/docs/tasks/repo2docker-local.qmd b/docs/tasks/repo2docker-local.qmd new file mode 100644 index 000000000..b23d423a6 --- /dev/null +++ b/docs/tasks/repo2docker-local.qmd @@ -0,0 +1,64 @@ +--- +title: Test User Images Locally +--- + +You should use `repo2docker` to build and test the image on your own device before you push and create a PR. It is often faster to do this first before using CI/CD since you can take advantage of local caching and rapid iteration. There's no need to waste Github Action minutes to test build images when you can do this on your own device. + +## Common Usage + +One can simply run `repo2docker /path/to/image/assets`. For example if one has changed into the directory containing the `repo2docker` files (such as `environment.yml` and/or `Dockerfile`), the command would be: + +```shell +repo2docker . +``` + +This works on Linux and Windows Subsystem for Linux (WSL). It will build the image, then launch jupyter server and display a localhost URL. Copy the URL and paste it into a local web browser. + +If you just want to build the image without also running the server, +add the `--no-run` argument: + +```shell +repo2docker --no-run . +``` + +## On Apple Silicon + +Apple's ARM-based CPUs (the "M" chips) are different from those run on the virtual machines in our clusters. macOS is capable of emulating x86_64/amd64, but it is necessary to optimize docker for this emulation, and to explicitly tell your local docker runtime that the images should be built on the `linux/amd64` platform. + +In Docker's settings: + + - Under **General** > **Virtual Machine Options**, either enable both **Apple Virtualization framework** and **Use Rosetta for x86_64/amd64 emulation on Apple Silicon**, or enable **Docker VMM**. + - Under **Resources** it is also recommended to raise the memory limit to at least 4GB. + +There are two methods for building `linux/amd64` images. The default uses `repo2docker`'s support for `docker-py`, while the second uses a `repo2docker` plugin that can invoke your local docker command-line interface. + +### docker-py (default) + +Run `jupyter-repo2docker` with the following arguments: + +``` +repo2docker \ + --Repo2Docker.platform=linux/amd64 \ + -e PLAYWRIGHT_BROWSERS_PATH=/srv/conda \ + --user-id=1000 --user-name=jovyan \ + --target-repo-dir=/home/jovyan/.cache \ + . +``` + +where the final parameter is the path to the assets or `.` if they are in the current directory. + +The `--user-id` and `--user-name` options are for non-Dockerfile based builds. Images with Dockerfiles do not need those options. + +Note that you may see (possibly harmless) architecture mismatch warnings with this method. + +### `docker` CLI + +You can instruct `repo2docker` to use your machine's local `docker` executable directly rather than the default of `docker-py`. You will first need to install [repo2podman](https://github.com/manics/repo2podman), a plugin that lets you use any container runtime with a command-line user interface similar to that of `docker`. This is useful if you want to leverage [docker buildx](https://github.com/docker/buildx/) (for things like multi-stage builds) or if you want to use an alternative executable like `podman`. This also eliminates architecture mismatch warnings. + +``` +repo2docker \ + --Repo2Docker.platform=linux/amd64 \ + -e PLAYWRIGHT_BROWSERS_PATH=/srv/conda \ + --engine podman --PodmanEngine.podman_executable=docker \ + . +``` From 12e8c0c73bf8753b40e7f7ec0524947f5c3b2df6 Mon Sep 17 00:00:00 2001 From: Ryan Lovett Date: Thu, 21 Nov 2024 10:03:59 -0800 Subject: [PATCH 2/3] Per feedback from @shaneknapp. --- docs/tasks/new-image.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tasks/new-image.qmd b/docs/tasks/new-image.qmd index e6b87aa6d..b0302cbcf 100644 --- a/docs/tasks/new-image.qmd +++ b/docs/tasks/new-image.qmd @@ -104,7 +104,7 @@ Create a PR and merge to staging. You can cancel the [`Deploy staging and prod hubs` job in Actions](https://github.com/berkeley-dsep-infra/datahub/actions/workflows/deploy-hubs.yaml), or just let it fail. -## Add a GitHub Slack Bot +## Subscribe to GitHub Repo in Slack Go to the #ucb-datahubs-bots channel, and run the following command: From 5368a6d7d05ec8b76125da72ea55df1d5b2758ee Mon Sep 17 00:00:00 2001 From: Ryan Lovett Date: Thu, 21 Nov 2024 10:11:30 -0800 Subject: [PATCH 3/3] Add warning about repo2podman and WSL. --- docs/tasks/repo2docker-local.qmd | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/tasks/repo2docker-local.qmd b/docs/tasks/repo2docker-local.qmd index b23d423a6..27b59a0bc 100644 --- a/docs/tasks/repo2docker-local.qmd +++ b/docs/tasks/repo2docker-local.qmd @@ -55,6 +55,10 @@ Note that you may see (possibly harmless) architecture mismatch warnings with th You can instruct `repo2docker` to use your machine's local `docker` executable directly rather than the default of `docker-py`. You will first need to install [repo2podman](https://github.com/manics/repo2podman), a plugin that lets you use any container runtime with a command-line user interface similar to that of `docker`. This is useful if you want to leverage [docker buildx](https://github.com/docker/buildx/) (for things like multi-stage builds) or if you want to use an alternative executable like `podman`. This also eliminates architecture mismatch warnings. +::: {.callout-warning} +repo2podman reportedly does not work yet on WSL. +::: + ``` repo2docker \ --Repo2Docker.platform=linux/amd64 \