From 64e2a1c2e39c64a3269014097f5bc43bfffb8669 Mon Sep 17 00:00:00 2001 From: ranko r sredojevic Date: Fri, 3 Apr 2020 14:21:10 -0700 Subject: [PATCH 1/7] Add instructions for CVAT over HTTPs. --- cvat/apps/documentation/installation.md | 295 ++++++++++++++++++++++++ 1 file changed, 295 insertions(+) diff --git a/cvat/apps/documentation/installation.md b/cvat/apps/documentation/installation.md index 1286fe813c2e..d6508a372159 100644 --- a/cvat/apps/documentation/installation.md +++ b/cvat/apps/documentation/installation.md @@ -7,6 +7,7 @@ - [Stop all containers](#stop-all-containers) - [Advanced settings](#advanced-settings) - [Share path](#share-path) + - [Serving over HTTPS](#serving-over-https) # Quick installation guide @@ -306,3 +307,297 @@ volumes: You can change the share device path to your actual share. For user convenience we have defined the environment variable $CVAT_SHARE_URL. This variable contains a text (url for example) which is shown in the client-share browser. + +### Serving over HTTPS + +We will add [letsencrypt.org](https://letsencrypt.org/) issued certificate to secure +our server connection. + +#### Prerequisites + +We assume that + + 1. you have sudo access on your server machine, + 2. you have an IP address to use for remote access, and + 2. that the local CVAT installation works on your server. + +If this is not the case, please complete the steps in the installation manual first. + +#### Roadmap + +We will go through the following sequence of steps to get CVAT over HTTPS: + + 1. Move Docker Compose CVAT access port to 80/tcp. + 2. Configure Nginx to serve the files to satisfy one of the [ACME challenges](https://letsencrypt.org/docs/challenge-types/). + 3. Create the certificate files using [acme.sh](https://github.com/acmesh-official/acme.sh). + 4. Reconfigure Nginx to serve over HTTPS and map CVAT to Docker Compose port 443. + +#### Step-by-step instructions + +##### 1. Move the CVAT access port + +Let's assume the server will be at `my-cvat-server.org`. + +```bash +# on the server +docker-compose down + +# add docker-compose.override.yml as per instructions below + +docker-compose up -d +``` + +Add the following into your `docker-compose.override.yml`, replacing `my-cvat-server.org` with your own IP address. +This file lives in the same directory as `docker-compose.yml`. + +```yaml +# docker-compose.override.yml +version: "2.3" + +services: + cvat_proxy: + environment: + CVAT_HOST: my-cvat-server.org + ports: + - "80:80" + + cvat: + environment: + ALLOWED_HOSTS: '*' +``` + +You should now see an unsecured version of CVAT at `http://my-cvat-server.org`. + +##### 2. Configure Nginx for the ACME challenge + +Temporarily, enable serving `http://my-cvat-server.org/.well-known/acme-challenge/` route from `/letsencrypt` directory on the server's filesystem. +You can use the [Nginx quickstart guide](http://nginx.org/en/docs/beginners_guide.html) for reference. + +```bash +# cvat_proxy/conf.d/cvat.conf.template + +server { + listen 80; + server_name _ default; + return 404; +} + +server { + listen 80; + server_name ${CVAT_HOST}; + + # add this temporarily, to pass an acme challenge + location ^~ /.well-known/acme-challenge/ { + allow all; + root /letsencrypt; + } + + location ~* /api/.*|git/.*|tensorflow/.*|auto_annotation/.*|analytics/.*|static/.*|admin|admin/.*|documentation/.*|dextr/.*|reid/.* { + proxy_pass http://cvat:8080; + proxy_pass_header X-CSRFToken; + proxy_set_header Host $http_host; + proxy_pass_header Set-Cookie; + } + + location / { + # workaround for match location by arguments + error_page 418 = @annotation_ui; + + if ( $query_string ~ "^id=\d+.*" ) { return 418; } + + proxy_pass http://cvat_ui; + proxy_pass_header X-CSRFToken; + proxy_set_header Host $http_host; + proxy_pass_header Set-Cookie; + } + + # old annotation ui, will be removed in the future. + location @annotation_ui { + proxy_pass http://cvat:8080; + proxy_pass_header X-CSRFToken; + proxy_set_header Host $http_host; + proxy_pass_header Set-Cookie; + } +} +``` + +Now create the `/letsencrypt` directory and mount it into `cvat_proxy` container. +Edit your `docker-compose.override.yml` to look like the following: + +```yaml +# docker-compose.override.yml +version: "2.3" + +services: + cvat_proxy: + environment: + CVAT_HOST: my-cvat-server.org + ports: + - "80:80" + volumes: + - ./letsencrypt:/letsencrypt + + cvat: + environment: + ALLOWED_HOSTS: '*' +``` + +Now create the directory and restart CVAT. + +```bash +# in the same directory where docker-compose.override.yml lives +mkdir -p letsencrypt/.well-known/acme-challenge + +docker-compose down +docker-compose up -d +``` + +Your server should still be visible (and unsecured) at `http://my-cvat-server.org` but you won't notice much chenge. + +##### 3. Create certificate files using an ACME challenge + +At this point your deployment is running. + +```bash +admin@tempVM:~/cvat$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +0a35cd127968 nginx:stable-alpine "/bin/sh -c 'envsubs…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, 0.0.0.0:8080->80/tcp cvat_proxy +b85497c44836 cvat_cvat_ui "nginx -g 'daemon of…" About a minute ago Up About a minute 80/tcp cvat_ui +d25a00475849 cvat "/usr/bin/supervisord" About a minute ago Up About a minute 8080/tcp, 8443/tcp cvat +6353a43f55c3 redis:4.0-alpine "docker-entrypoint.s…" About a minute ago Up About a minute 6379/tcp cvat_redis +52009636caa8 postgres:10-alpine "docker-entrypoint.s…" About a minute ago Up About a minute 5432/tcp cvat_db +``` + +Now we will attach `cvat_proxy` container to run `acme.sh` scripts. + +```bash +admin@tempVM:~/cvat$ docker exec -ti cvat_proxy /bin/sh + +# now we install some missing software inside cvat_proxy +/ # apk add openssl curl +/ # curl https://get.acme.sh | sh +/ # ~/.acme.sh/acme.sh -h +[... many lines ...] + +/ # ~/.acme.sh/acme.sh --issue -d my-cvat-server.org -w /letsencrypt +[Fri Apr 3 20:49:05 UTC 2020] Create account key ok. +[Fri Apr 3 20:49:05 UTC 2020] Registering account +[Fri Apr 3 20:49:06 UTC 2020] Registered +[Fri Apr 3 20:49:06 UTC 2020] ACCOUNT_THUMBPRINT='tril8-LdJgM8xg6mnN1pMa7vIMdFizVCE0NImNmyZY4' +[Fri Apr 3 20:49:06 UTC 2020] Creating domain key +[ ... many more lines ...] +[Fri Apr 3 20:49:10 UTC 2020] Your cert is in /root/.acme.sh/my-cvat-server.org/my-cvat-server.org.cer +[Fri Apr 3 20:49:10 UTC 2020] Your cert key is in /root/.acme.sh/my-cvat-server.org/my-cvat-server.org.key +[Fri Apr 3 20:49:10 UTC 2020] The intermediate CA cert is in /root/.acme.sh/my-cvat-server.org/ca.cer +[Fri Apr 3 20:49:10 UTC 2020] And the full chain certs is there: /root/.acme.sh/my-cvat-server.org/fullchain.cer + +/ # cp ~/.acme.sh/my-cvat-server.org/my-cvat-server.org.cer /letsencrypt/certificate.cer +/ # cp ~/.acme.sh/my-cvat-server.org/my-cvat-server.org.key /letsencrypt/certificate.key +/ # cp ~/.acme.sh/my-cvat-server.org/ca.cer /letsencrypt/ca.cer +/ # cp ~/.acme.sh/my-cvat-server.org/fullchain.cer /letsencrypt/fullchain.cer +/ # exit +admin@tempVM:~/cvat$ ls letsencrypt/ +ca.cer certificate.cer certificate.key fullchain.cer +admin@tempVM:~/cvat$ mkdir cert +admin@tempVM:~/cvat$ mv letsencrypt/* ./cert +``` + +##### 4. Reconfigure Nginx for HTTPS access + +Update Docker Compose configuration to mount the certificate directory. + +```yml +# docker-compose.override.yml +version: "2.3" + +services: + cvat_proxy: + environment: + CVAT_HOST: my-cvat-server.westus2.cloudapp.azure.com + ports: + - "443:443" + volumes: + - ./letsencrypt:/letsencrypt + - ./cert:/cert:ro + + cvat: + environment: + ALLOWED_HOSTS: '*' +``` + +Also, reconfigure Nginx to use `443/tcp` and point it to the new keys. + +```bash +server { + listen 80; + server_name _ default; + return 404; +} + +server { + listen 443 ssl; + server_name ${CVAT_HOST}; + ssl_certificate /cert/certificate.cer; + ssl_certificate_key /cert/certificate.key; + + location ~* /api/.*|git/.*|tensorflow/.*|auto_annotation/.*|analytics/.*|static/.*|admin|admin/.*|documentation/.*|dextr/.*|reid/.* { + proxy_pass http://cvat:8080; + proxy_pass_header X-CSRFToken; + proxy_set_header Host $http_host; + proxy_pass_header Set-Cookie; + } + + location / { + # workaround for match location by arguments + error_page 418 = @annotation_ui; + + if ( $query_string ~ "^id=\d+.*" ) { return 418; } + + proxy_pass http://cvat_ui; + proxy_pass_header X-CSRFToken; + proxy_set_header Host $http_host; + proxy_pass_header Set-Cookie; + } + + # old annotation ui, will be removed in the future. + location @annotation_ui { + proxy_pass http://cvat:8080; + proxy_pass_header X-CSRFToken; + proxy_set_header Host $http_host; + proxy_pass_header Set-Cookie; + } +} +``` + +Finally, restart your service. + +```bash +admin@tempVM:~/cvat$ docker-compose down +Stopping cvat_proxy ... done +Stopping cvat_ui ... done +Stopping cvat ... done +Stopping cvat_db ... done +Stopping cvat_redis ... done +Removing cvat_proxy ... done +Removing cvat_ui ... done +Removing cvat ... done +Removing cvat_db ... done +Removing cvat_redis ... done +Removing network cvat_default +admin@tempVM:~/cvat$ docker-compose up -d +Creating network "cvat_default" with the default driver +Creating cvat_db ... done +Creating cvat_redis ... done +Creating cvat ... done +Creating cvat_ui ... done +Creating cvat_proxy ... done +super@tempVM:~/cvat$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +71464aeac87c nginx:stable-alpine "/bin/sh -c 'envsubs…" About a minute ago Up About a minute 0.0.0.0:443->443/tcp, 0.0.0.0:8080->80/tcp cvat_proxy +8428cfbb766e cvat_cvat_ui "nginx -g 'daemon of…" About a minute ago Up About a minute 80/tcp cvat_ui +b5a2f78689da cvat "/usr/bin/supervisord" About a minute ago Up About a minute 8080/tcp, 8443/tcp cvat +ef4a1f47440f redis:4.0-alpine "docker-entrypoint.s…" About a minute ago Up About a minute 6379/tcp cvat_redis +7803bf828d9f postgres:10-alpine "docker-entrypoint.s…" About a minute ago Up About a minute 5432/tcp cvat_db +``` + +Now you can go to `https://my-cvat-server.org/` and verify that you are using an encrypted connection. From 870ab873bb6875f4e3af1fce1c393943561a2e1a Mon Sep 17 00:00:00 2001 From: ranko r sredojevic Date: Fri, 3 Apr 2020 14:25:55 -0700 Subject: [PATCH 2/7] Fix typos in CVAT+HTTPS docs. --- cvat/apps/documentation/installation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cvat/apps/documentation/installation.md b/cvat/apps/documentation/installation.md index d6508a372159..664f1cd672d4 100644 --- a/cvat/apps/documentation/installation.md +++ b/cvat/apps/documentation/installation.md @@ -328,7 +328,7 @@ If this is not the case, please complete the steps in the installation manual fi We will go through the following sequence of steps to get CVAT over HTTPS: 1. Move Docker Compose CVAT access port to 80/tcp. - 2. Configure Nginx to serve the files to satisfy one of the [ACME challenges](https://letsencrypt.org/docs/challenge-types/). + 2. Configure Nginx to pass one of the [ACME challenges](https://letsencrypt.org/docs/challenge-types/). 3. Create the certificate files using [acme.sh](https://github.com/acmesh-official/acme.sh). 4. Reconfigure Nginx to serve over HTTPS and map CVAT to Docker Compose port 443. @@ -452,7 +452,7 @@ docker-compose down docker-compose up -d ``` -Your server should still be visible (and unsecured) at `http://my-cvat-server.org` but you won't notice much chenge. +Your server should still be visible (and unsecured) at `http://my-cvat-server.org` but you won't any behavior changes. ##### 3. Create certificate files using an ACME challenge @@ -591,7 +591,7 @@ Creating cvat_redis ... done Creating cvat ... done Creating cvat_ui ... done Creating cvat_proxy ... done -super@tempVM:~/cvat$ docker ps +admin@tempVM:~/cvat$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 71464aeac87c nginx:stable-alpine "/bin/sh -c 'envsubs…" About a minute ago Up About a minute 0.0.0.0:443->443/tcp, 0.0.0.0:8080->80/tcp cvat_proxy 8428cfbb766e cvat_cvat_ui "nginx -g 'daemon of…" About a minute ago Up About a minute 80/tcp cvat_ui From 8811861dd2bb8ee386fa348b9f14f98bf9ad965f Mon Sep 17 00:00:00 2001 From: ranko r sredojevic Date: Sat, 4 Apr 2020 13:51:48 -0700 Subject: [PATCH 3/7] Clean up CVAT+HTTPS section for style compliance. --- cvat/apps/documentation/installation.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/cvat/apps/documentation/installation.md b/cvat/apps/documentation/installation.md index 664f1cd672d4..03c76a95a1f1 100644 --- a/cvat/apps/documentation/installation.md +++ b/cvat/apps/documentation/installation.md @@ -317,9 +317,9 @@ our server connection. We assume that - 1. you have sudo access on your server machine, - 2. you have an IP address to use for remote access, and - 2. that the local CVAT installation works on your server. +1. you have sudo access on your server machine, +2. you have an IP address to use for remote access, and +3. that the local CVAT installation works on your server. If this is not the case, please complete the steps in the installation manual first. @@ -327,10 +327,10 @@ If this is not the case, please complete the steps in the installation manual fi We will go through the following sequence of steps to get CVAT over HTTPS: - 1. Move Docker Compose CVAT access port to 80/tcp. - 2. Configure Nginx to pass one of the [ACME challenges](https://letsencrypt.org/docs/challenge-types/). - 3. Create the certificate files using [acme.sh](https://github.com/acmesh-official/acme.sh). - 4. Reconfigure Nginx to serve over HTTPS and map CVAT to Docker Compose port 443. +1. Move Docker Compose CVAT access port to 80/tcp. +2. Configure Nginx to pass one of the [ACME challenges](https://letsencrypt.org/docs/challenge-types/). +3. Create the certificate files using [acme.sh](https://github.com/acmesh-official/acme.sh). +4. Reconfigure Nginx to serve over HTTPS and map CVAT to Docker Compose port 443. #### Step-by-step instructions @@ -442,7 +442,7 @@ services: ALLOWED_HOSTS: '*' ``` -Now create the directory and restart CVAT. +Finally, create the directory and restart CVAT. ```bash # in the same directory where docker-compose.override.yml lives @@ -452,7 +452,7 @@ docker-compose down docker-compose up -d ``` -Your server should still be visible (and unsecured) at `http://my-cvat-server.org` but you won't any behavior changes. +Your server should still be visible (and unsecured) at `http://my-cvat-server.org` but you won't see any behavior changes. ##### 3. Create certificate files using an ACME challenge @@ -468,12 +468,12 @@ d25a00475849 cvat "/usr/bin/supervisord" About a minut 52009636caa8 postgres:10-alpine "docker-entrypoint.s…" About a minute ago Up About a minute 5432/tcp cvat_db ``` -Now we will attach `cvat_proxy` container to run `acme.sh` scripts. +We will attach `cvat_proxy` container to run `acme.sh` scripts. ```bash admin@tempVM:~/cvat$ docker exec -ti cvat_proxy /bin/sh -# now we install some missing software inside cvat_proxy +# install some missing software inside cvat_proxy / # apk add openssl curl / # curl https://get.acme.sh | sh / # ~/.acme.sh/acme.sh -h @@ -518,7 +518,7 @@ services: - "443:443" volumes: - ./letsencrypt:/letsencrypt - - ./cert:/cert:ro + - ./cert:/cert:ro # this is new cvat: environment: From 106e4b756b60c39bcbd7607ba72abe50d7cb25b2 Mon Sep 17 00:00:00 2001 From: ranko r sredojevic Date: Sun, 5 Apr 2020 15:10:29 -0700 Subject: [PATCH 4/7] Format lists in Installation.md (https subsection). --- cvat/apps/documentation/installation.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/cvat/apps/documentation/installation.md b/cvat/apps/documentation/installation.md index 03c76a95a1f1..6cc5ad81767f 100644 --- a/cvat/apps/documentation/installation.md +++ b/cvat/apps/documentation/installation.md @@ -317,9 +317,9 @@ our server connection. We assume that -1. you have sudo access on your server machine, -2. you have an IP address to use for remote access, and -3. that the local CVAT installation works on your server. +- you have sudo access on your server machine, +- you have an IP address to use for remote access, and +- that the local CVAT installation works on your server. If this is not the case, please complete the steps in the installation manual first. @@ -327,10 +327,10 @@ If this is not the case, please complete the steps in the installation manual fi We will go through the following sequence of steps to get CVAT over HTTPS: -1. Move Docker Compose CVAT access port to 80/tcp. -2. Configure Nginx to pass one of the [ACME challenges](https://letsencrypt.org/docs/challenge-types/). -3. Create the certificate files using [acme.sh](https://github.com/acmesh-official/acme.sh). -4. Reconfigure Nginx to serve over HTTPS and map CVAT to Docker Compose port 443. +- Move Docker Compose CVAT access port to 80/tcp. +- Configure Nginx to pass one of the [ACME challenges](https://letsencrypt.org/docs/challenge-types/). +- Create the certificate files using [acme.sh](https://github.com/acmesh-official/acme.sh). +- Reconfigure Nginx to serve over HTTPS and map CVAT to Docker Compose port 443. #### Step-by-step instructions @@ -370,7 +370,8 @@ You should now see an unsecured version of CVAT at `http://my-cvat-server.org`. ##### 2. Configure Nginx for the ACME challenge -Temporarily, enable serving `http://my-cvat-server.org/.well-known/acme-challenge/` route from `/letsencrypt` directory on the server's filesystem. +Temporarily, enable serving `http://my-cvat-server.org/.well-known/acme-challenge/` +route from `/letsencrypt` directory on the server's filesystem. You can use the [Nginx quickstart guide](http://nginx.org/en/docs/beginners_guide.html) for reference. ```bash @@ -452,7 +453,8 @@ docker-compose down docker-compose up -d ``` -Your server should still be visible (and unsecured) at `http://my-cvat-server.org` but you won't see any behavior changes. +Your server should still be visible (and unsecured) at `http://my-cvat-server.org` +but you won't see any behavior changes. ##### 3. Create certificate files using an ACME challenge @@ -513,7 +515,7 @@ version: "2.3" services: cvat_proxy: environment: - CVAT_HOST: my-cvat-server.westus2.cloudapp.azure.com + CVAT_HOST: my-cvat-server.org ports: - "443:443" volumes: From af82589a184dd95193ac5a0d83d1a0b0469f4abc Mon Sep 17 00:00:00 2001 From: ranko r sredojevic Date: Sun, 5 Apr 2020 15:17:48 -0700 Subject: [PATCH 5/7] Tweak spaces in bulleted lists. --- cvat/apps/documentation/installation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cvat/apps/documentation/installation.md b/cvat/apps/documentation/installation.md index 6cc5ad81767f..ca508e9e267b 100644 --- a/cvat/apps/documentation/installation.md +++ b/cvat/apps/documentation/installation.md @@ -317,9 +317,9 @@ our server connection. We assume that -- you have sudo access on your server machine, -- you have an IP address to use for remote access, and -- that the local CVAT installation works on your server. +- you have sudo access on your server machine, +- you have an IP address to use for remote access, and +- that the local CVAT installation works on your server. If this is not the case, please complete the steps in the installation manual first. From 46fbc2ca503cfabfdbc252e582e927e2b9f6500f Mon Sep 17 00:00:00 2001 From: ranko r sredojevic Date: Sun, 5 Apr 2020 15:42:09 -0700 Subject: [PATCH 6/7] Update CHANGELOG with info about CVAT+HTTPS tutorial. --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 52efb211e13f..c0fefe029336 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.6.2] - 2020-04-05 +### Changed +- Added instructions for CVAT over HTTPS to documentation. + ## [0.6.1] - 2020-03-21 ### Changed - VOC task export now does not use official label map by default, but takes one From a13b1591882c98ae6e8ab3f63616c535def02704 Mon Sep 17 00:00:00 2001 From: ranko r sredojevic Date: Sat, 11 Apr 2020 09:59:00 -0700 Subject: [PATCH 7/7] Fix CHANGELOG comment for HTTPS instructions to 1.0.0-beta. --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c0fefe029336..1ca4e1591e62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.6.2] - 2020-04-05 +## [1.0.0-beta] - 2020-04-05 ### Changed - Added instructions for CVAT over HTTPS to documentation.