Could not get models from the server

[beep-love]

My actions before raising this issue

• Hi, I was following the instructions as per https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/documentation/installation.md .

• As per the document I was able to compose docker for CVAT and the tool was successfully hosted at http://localhost:8080/

• After the deployment, i was also able to compose a serverless component from docker using

~/cvat$ docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d --build

The above steps recreated cvat and nuclio. The log is provided below

citadmin@citadmin-MS-7B17:~/cvat$ docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d --build
WARNING: The no_proxy variable is not set. Defaulting to a blank string.
Building cvat
Step 1/34 : FROM ubuntu:20.04
 ---> f643c72bc252
Step 2/34 : ARG http_proxy
 ---> Using cache
 ---> 37786d23c823
Step 3/34 : ARG https_proxy
 ---> Using cache
 ---> 2d3a11ddd1ee
Step 4/34 : ARG no_proxy
 ---> Using cache
 ---> 01107e9eaab6
Step 5/34 : ARG socks_proxy
 ---> Using cache
 ---> 16fdaa939927
Step 6/34 : ARG TZ
 ---> Using cache
 ---> 0cea62edde92
Step 7/34 : ENV TERM=xterm     http_proxy=${http_proxy}       https_proxy=${https_proxy}     no_proxy=${no_proxy}     socks_proxy=${socks_proxy}     LANG='C.UTF-8'      LC_ALL='C.UTF-8'     TZ=${TZ}
 ---> Using cache
 ---> e6310d2dcc16
Step 8/34 : ARG USER
 ---> Using cache
 ---> 566c2a77ef34
Step 9/34 : ARG DJANGO_CONFIGURATION
 ---> Using cache
 ---> 4e33706e112c
Step 10/34 : ENV DJANGO_CONFIGURATION=${DJANGO_CONFIGURATION}
 ---> Using cache
 ---> 950f302818a5
Step 11/34 : RUN apt-get update &&     apt-get --no-install-recommends install -yq         software-properties-common &&     apt-get update &&     DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends install -yq         apache2         apache2-dev         apt-utils         build-essential         libapache2-mod-xsendfile         supervisor         libavcodec-dev=7:4.2.4-1ubuntu0.1         libavdevice-dev=7:4.2.4-1ubuntu0.1         libavfilter-dev=7:4.2.4-1ubuntu0.1         libavformat-dev=7:4.2.4-1ubuntu0.1         libavutil-dev=7:4.2.4-1ubuntu0.1         libswresample-dev=7:4.2.4-1ubuntu0.1         libswscale-dev=7:4.2.4-1ubuntu0.1         libldap2-dev         libsasl2-dev         pkg-config         python3-dev         python3-pip         tzdata         p7zip-full         git         git-lfs         ssh         poppler-utils         curl &&     python3 -m pip install --no-cache-dir -U pip==20.0.1 setuptools==49.6.0 wheel==0.35.1 &&     ln -fs /usr/share/zoneinfo/${TZ} /etc/localtime &&     dpkg-reconfigure -f noninteractive tzdata &&     rm -rf /var/lib/apt/lists/* &&     echo 'application/wasm wasm' >> /etc/mime.types
 ---> Using cache
 ---> 1424b0bf7592
Step 12/34 : ENV USER=${USER}
 ---> Using cache
 ---> 2dc37e4c3341
Step 13/34 : ENV HOME /home/${USER}
 ---> Using cache
 ---> 055c8556c3a1
Step 14/34 : WORKDIR ${HOME}
 ---> Using cache
 ---> 34d655954553
Step 15/34 : RUN adduser --shell /bin/bash --disabled-password --gecos "" ${USER} &&     if [ -z ${socks_proxy} ]; then         echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30\"" >> ${HOME}/.bashrc;     else         echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30 -o ProxyCommand='nc -X 5 -x ${socks_proxy} %h %p'\"" >> ${HOME}/.bashrc;     fi
 ---> Using cache
 ---> 2dc0a4d111ce
Step 16/34 : COPY components /tmp/components
 ---> Using cache
 ---> 85d89ddfb936
Step 17/34 : COPY cvat/requirements/ /tmp/requirements/
 ---> Using cache
 ---> 76e738457a85
Step 18/34 : COPY supervisord.conf mod_wsgi.conf wait-for-it.sh manage.py ${HOME}/
 ---> Using cache
 ---> 946450363ea5
Step 19/34 : RUN python3 -m pip install --no-cache-dir -r /tmp/requirements/${DJANGO_CONFIGURATION}.txt
 ---> Using cache
 ---> 7b47cd56b7fd
Step 20/34 : ARG CLAM_AV
 ---> Using cache
 ---> 43e5df864d60
Step 21/34 : ENV CLAM_AV=${CLAM_AV}
 ---> Using cache
 ---> 3e6f4ffa2ac3
Step 22/34 : RUN if [ "$CLAM_AV" = "yes" ]; then         apt-get update &&         apt-get --no-install-recommends install -yq             clamav             libclamunrar9 &&         sed -i 's/ReceiveTimeout 30/ReceiveTimeout 300/g' /etc/clamav/freshclam.conf &&         freshclam &&         chown -R ${USER}:${USER} /var/lib/clamav &&         rm -rf /var/lib/apt/lists/*;     fi
 ---> Using cache
 ---> 849534334192
Step 23/34 : COPY ssh ${HOME}/.ssh
 ---> Using cache
 ---> 5c8a7c951675
Step 24/34 : COPY utils ${HOME}/utils
 ---> Using cache
 ---> 24c2beff1047
Step 25/34 : COPY cvat/ ${HOME}/cvat
 ---> Using cache
 ---> 0b2f35e80b50
Step 26/34 : COPY cvat-core/ ${HOME}/cvat-core
 ---> Using cache
 ---> d76206c76bb7
Step 27/34 : COPY cvat-data/ ${HOME}/cvat-data
 ---> Using cache
 ---> e4f8b0691120
Step 28/34 : COPY tests ${HOME}/tests
 ---> Using cache
 ---> 3a814f3be705
Step 29/34 : RUN chown -R ${USER}:${USER} .
 ---> Using cache
 ---> d68939f567a3
Step 30/34 : USER ${USER}
 ---> Using cache
 ---> e1de77d28fdd
Step 31/34 : RUN mkdir data share media keys logs /tmp/supervisord
 ---> Using cache
 ---> 373bd2662de6
Step 32/34 : RUN python3 manage.py collectstatic
 ---> Using cache
 ---> 2fc80ebab664
Step 33/34 : EXPOSE 8080 8443
 ---> Using cache
 ---> 7e18d295d278
Step 34/34 : ENTRYPOINT ["/usr/bin/supervisord"]
 ---> Using cache
 ---> c0e0ad888f99

Successfully built c0e0ad888f99
Successfully tagged cvat/server:latest
Building cvat_ui
Step 1/29 : FROM node:lts-alpine AS cvat-ui
 ---> 7f1893c3ede0
Step 2/29 : ARG http_proxy
 ---> Using cache
 ---> cfe7bf23d890
Step 3/29 : ARG https_proxy
 ---> Using cache
 ---> 9d5a585b4096
Step 4/29 : ARG no_proxy
 ---> Using cache
 ---> 0d5b54e97089
Step 5/29 : ARG socks_proxy
 ---> Using cache
 ---> fd47b78b0715
Step 6/29 : ARG PUBLIC_INSTANCE
 ---> Using cache
 ---> 966984d2bf15
Step 7/29 : ARG WA_PAGE_VIEW_HIT
 ---> Using cache
 ---> a5b3a0b5f9ff
Step 8/29 : ENV TERM=xterm     http_proxy=${http_proxy}       https_proxy=${https_proxy}     no_proxy=${no_proxy}     socks_proxy=${socks_proxy}     LANG='C.UTF-8'      LC_ALL='C.UTF-8'
 ---> Using cache
 ---> da3a720116dd
Step 9/29 : COPY cvat-core/package*.json /tmp/cvat-core/
 ---> Using cache
 ---> aee3a8734740
Step 10/29 : COPY cvat-canvas/package*.json /tmp/cvat-canvas/
 ---> Using cache
 ---> 5d08a513756a
Step 11/29 : COPY cvat-ui/package*.json /tmp/cvat-ui/
 ---> Using cache
 ---> e0ff4a2d07a9
Step 12/29 : COPY cvat-data/package*.json /tmp/cvat-data/
 ---> Using cache
 ---> a1ed3afdd21e
Step 13/29 : WORKDIR /tmp/cvat-data/
 ---> Using cache
 ---> a109eddc31d2
Step 14/29 : RUN npm ci
 ---> Using cache
 ---> 253a387bbc59
Step 15/29 : WORKDIR /tmp/cvat-core/
 ---> Using cache
 ---> 047772d0fe11
Step 16/29 : RUN npm ci
 ---> Using cache
 ---> d978ec819a94
Step 17/29 : WORKDIR /tmp/cvat-canvas/
 ---> Using cache
 ---> b3a5c405c264
Step 18/29 : RUN npm ci
 ---> Using cache
 ---> 2e7f4733f079
Step 19/29 : WORKDIR /tmp/cvat-ui/
 ---> Using cache
 ---> 6cc3225c3074
Step 20/29 : RUN npm ci
 ---> Using cache
 ---> 9b81141e8127
Step 21/29 : COPY cvat-data/ /tmp/cvat-data/
 ---> Using cache
 ---> 6ddb2c06df1a
Step 22/29 : COPY cvat-core/ /tmp/cvat-core/
 ---> Using cache
 ---> e64890b0a21a
Step 23/29 : COPY cvat-canvas/ /tmp/cvat-canvas/
 ---> Using cache
 ---> e6b61df25cc8
Step 24/29 : COPY cvat-ui/ /tmp/cvat-ui/
 ---> Using cache
 ---> e4c8a7ab61b0
Step 25/29 : RUN npm run build
 ---> Using cache
 ---> 361ef88e1a2e

Step 26/29 : FROM nginx:stable-alpine
 ---> a8054a07a2ce
Step 27/29 : RUN sed -i "s/}/application\/wasm wasm;\n}/g" /etc/nginx/mime.types
 ---> Using cache
 ---> 158c07470d96
Step 28/29 : COPY cvat-ui/react_nginx.conf /etc/nginx/conf.d/default.conf
 ---> Using cache
 ---> c8cb20a6cf64
Step 29/29 : COPY --from=cvat-ui /tmp/cvat-ui/dist /usr/share/nginx/html/
 ---> Using cache
 ---> ccca981898db

Successfully built ccca981898db
Successfully tagged cvat/ui:latest
cvat_db is up-to-date
cvat_redis is up-to-date
Recreating nuclio ... done
Recreating cvat   ... done
cvat_ui is up-to-date
cvat_proxy is up-to-date

• I was also able to deploy nuclio functions for semi-automatic annotations. After deploying nuclio, i was able to see functions at http://localhost:8070/projects/cvat/functions

Current Behaviour

After above-mentioned steps, when I went to http://localhost:8080/models, the browser was not able to load the models and displayed the error at the web console:

GET http://localhost:8080/api/v1/lambda/functions 500 (Internal xhr.js:184 GET http://localhost:8080/api/v1/lambda/functions 500 (Internal Server Error)

Context

I was trying to run auto annotation using available models. But the problem in my view seems to be API error. Also, i have skipped the advanced topic: Deploying CVAT behind the proxy. Does that make any issues related to my problem??

Environment

• Docker version 19.03.14, build 5eb3275d40
• using Docker Swarm
• Operating System: Ubuntu 20.04.1 LTS

Next steps

You may join our Gitter channel for community support.

[azhavoro]

@beep-love Hi, please attach logs from cvat container: docker logs cvat > cvat.log

[beep-love]

PFA

cvat.log

[beep-love]

I changed the line since I am using version nuclio 1.5.7

and rebuild the composer.

I have attached the updated log file after doing this.
cvat.log

[jahaniam]

GET http://localhost:8080/api/v1/lambda/functions 500 (Internal xhr.js:184 GET http://localhost:8080/api/v1/lambda/functions 500 (Internal Server Error)

Your error happens within the function.
do the docker ps -a and find the container that is related to it. Should start with: nuclio-nuclio-....
then do docker log <NAME> you will see the actual error.

[beep-love]

GET http://localhost:8080/api/v1/lambda/functions 500 (Internal xhr.js:184 GET http://localhost:8080/api/v1/lambda/functions 500 (Internal Server Error)

Your error happens within the function.
do the docker ps -a and find the container that is related to it. Should start with: nuclio-nuclio-....
then do docker log <NAME> you will see the actual error.

[beep-love]

@jahaniam Thanks for the help. It worked after deleting the functions from nuclio dashboard before rebuilding the function. I was able to load models and auto annotate with faster RCNN function.

Regards

[Erdos1729]

@beep-love can you please tell me how you rebuilt the function? I am facing a same issue of "Could not get models from the server"

[beep-love]

Use this

docker-compose down --remove-orphan

chmod +x nuctl-1.5.8 after downloading the nuctl

nuctl-1.5.8 deploy tf-faster-rcnn-inception-v2-coco-gpu \
--project-name cvat --path "serverless/tensorflow/faster_rcnn_inception_v2_coco/nuclio" --platform local \
--base-image tensorflow/tensorflow:2.1.1-gpu \
--desc "Faster RCNN from Tensorflow Object Detection GPU API" \
--image cvat/tf.faster_rcnn_inception_v2_coco_gpu \
--resource-limit nvidia.com/gpu=1

[beep-love]

Also, delete the nuclio error function from nuclio dashboard!