Unable to reach CVAT running on AWS EC2 from outside

[MahdiEsf]

I am using the most recent docker-compose file to build images including cvat_proxy server which exposes port 8080 (0.2.0-503-gec2fa6e).
Containers are running successfully on EC2 server to which ELB is attached and checks health on port 8080. There are target group and security group in place as well that control traffic.

Docker-compose file is the same as the new default version with the only difference that "CVAT_HOST" was changed to ".amazonaws.com" from "localhost" (Previous AWS deployment instruction to create docker-compose.override.yml is not applicable to new docker-compose which runs CVAT on single port 8080)

Here are the runnings containers:

When I curl ELB DNS like "curl -ivk ELB_DNS" from inside the server, code 200 is returned as following meaning I can reach CVAT from inside:

But when I try to get to CVAT from outside, i.e., local machine, it fails with code 404. The following is the logs of nginx container:

The green highlighted line is the curl output from inside server (code 200) whereas the other ones are from outside associated with ELB with 404 code. Have tried CVAT_HOST with EC2 IP address, ELB DNS, ELB IP but none of them worked out to reach to CVAT login page.

Any ideas/suggestions are appreciated.

Here are the contents of my docker-compose.yml file:

services:
cvat_db:
container_name: cvat_db
image: postgres:10-alpine
networks:
default:
aliases:
- db
restart: always
environment:
POSTGRES_USER: root
POSTGRES_DB: cvat
POSTGRES_HOST_AUTH_METHOD: trust
volumes:
- cvat_db:/var/lib/postgresql/data

cvat_redis:
container_name: cvat_redis
image: redis:4.0-alpine
networks:
default:
aliases:
- redis
restart: always

cvat:
container_name: cvat
image: cvat
restart: always
depends_on:
- cvat_redis
- cvat_db
build:
context: .
args:
http_proxy:
https_proxy:
no_proxy:
socks_proxy:
TF_ANNOTATION: "no"
AUTO_SEGMENTATION: "no"
USER: "django"
DJANGO_CONFIGURATION: "production"
TZ: "Etc/UTC"
OPENVINO_TOOLKIT: "no"
environment:
DJANGO_MODWSGI_EXTRA_ARGS: ""
ALLOWED_HOSTS: '*'
volumes:
- cvat_data:/home/django/data
- cvat_keys:/home/django/keys
- cvat_logs:/home/django/logs
- cvat_models:/home/django/models

cvat_ui:
container_name: cvat_ui
restart: always
build:
context: .
args:
http_proxy:
https_proxy:
no_proxy:
socks_proxy:
dockerfile: Dockerfile.ui

networks:
  default:
    aliases:
      - ui
depends_on:
  - cvat

cvat_proxy:
container_name: cvat_proxy
image: nginx:stable-alpine
restart: always
depends_on:
- cvat
- cvat_ui
environment:
CVAT_HOST: .amazonaws.com
ports:
- "8080:80"
volumes:
- ./cvat_proxy/nginx.conf:/etc/nginx/nginx.conf:ro
- ./cvat_proxy/conf.d/cvat.conf.template:/etc/nginx/conf.d/cvat.conf.template:ro
command: /bin/sh -c "envsubst '$$CVAT_HOST' < /etc/nginx/conf.d/cvat.conf.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"

volumes:
cvat_db:
cvat_data:
cvat_keys:
cvat_logs:
cvat_models:

[azhavoro]

Hi, thanks for the report. At the moment I have no idea why proxy doesn't work properly with public ip or FQDN, let me some time to deploy CVAT aws and check it.

[Onion-Skins]

Hi MahdiEsf, I just deployed CVAT successfully on a cloud VM (not AWS though). I just follow the instruction in the installation guide by creating docker-compose.override.yml with this content:
version: "2.3"

services:
cvat_proxy:
environment:
CVAT_HOST: .example.com # change this to your server's FQDN or ip address

In your config, ".amazonaws.com" is not a FQDN. I guess that's the cause of the issue.

[nmanovic]

@nmanovic , I have found two issues with the deployment:

• nginx proxy cannot work by default with long domain names (e.g. ec2-x-xx-xx-xxxx.us-east-2.compute.amazonaws.com). Need to specify a directive to resolve the issue. If you have the problem it is easy do diagnose. Just run docker logs cvat_proxy and you will see the recommendation from nginx.
• When cvat-ui is complied there is a problem with memory consumed by nodejs. Some problems can be solved by use NODE_OPTIONS=--max_old_space_size=200 environment variable. But webpack consumes too much memory. It leads to "JavaScript heap out of memory" problem (Using webpack 4 on a large project (or, how to avoid "JavaScript heap out of memory" with production mode) webpack/webpack#6389).

Our team will do the best to resolve the issue ASAP.

P.S. I tried to reproduce the issue on t2.micro with 1G RAM.

> webpack --profile --json --config ./webpack.config.js

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200324.204721.16.0.001.json
Node.js report completed

<--- Last few GCs --->

[16:0x55f791291ea0]   507863 ms: Mark-sweep 492.9 (500.8) -> 491.2 (497.1) MB, 10756.2 / 0.0 ms  (+ 375.1 ms in 55 steps since start of marking, biggest step 162.6 ms, walltime since start of marking 11945 ms) (average mu = 0.099, current mu = 0.068) allo[16:0x55f791291ea0]   519641 ms: Mark-sweep 493.2 (497.1) -> 491.4 (496.3) MB, 11163.4 / 0.0 ms  (+ 210.5 ms in 32 steps since start of marking, biggest step 205.8 ms, walltime since start of marking 11778 ms) (average mu = 0.067, current mu = 0.034) allo

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x55f78f2dfed9]
Security context: 0x0c3b661408d1 <JSObject>
    1: _(aka _) [0x350aa300d911] [/tmp/cvat-ui/node_modules/terser/dist/bundle.min.js:~1] [pc=0x84c1be5aac9](this=0x102210b404b1 <undefined>,0x0a1dc25d2239 <String[#4]: punc>,0x230c7833c009 <String[#1]: )>,0x102210b404b1 <undefined>)
    2: ee [0x3e8211e618f9] [/tmp/cvat-ui/node_modules/terser/dist/bundle.min.js:~1] [pc=0x84c1be0e1ae](this=0x0ad44b30a1a1 <Ob...

Removing intermediate container dcb710322576
 ---> b0ccebd8ec18