nuctl (nuclio) fails to deploy serverless functions

[inecho]

My actions before raising this issue

• Read/searched the docs
• Searched past
• Installation Auto Annotation
• Unable enable Semi-automatic and automatic annotation #2605
• installation_automatic_annotation.md guide needs to update to reflect the path change in develop branch #2843
• Tutorial about serverless functions #3124
• Couldn't create nuclio functions #2067
• Restarting nuclio problem nuclio/nuclio#2112 (comment)
• Use docker volume to mount the processor config to function container nuclio/nuclio#1820 (comment)

Expected Behaviour

I don't know how exactly it should look like, but at least i expect to have a successfully deployed model in nuclio at localhost:8070, so i can choose one in my CVAT tasks.

Current Behaviour

It seems like it's not able to find some pre-built folders in /tmp/ of the composed cvat docker images. Since im new to both frameworks, cvat and nuclio, i don't know how to properly debug this. What i did, was to follow the docs and serverless tutorial PR closely, shut the images down, rebuilt them, even on another machine with ubuntu 18.04 (my main has already 20.04), but the error stays the same. This holds for all functions in cvat/serverless/.

Input from cvat root folder:

nuctl deploy --project-name cvat \
  --path serverless/openvino/dextr/nuclio \
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

Output:

21.07.20 11:44:06.249                     nuctl (I) Deploying function {"name": ""}
21.07.20 11:44:06.249                     nuctl (I) Building {"versionInfo": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
21.07.20 11:44:06.650                     nuctl (I) Cleaning up before deployment {"functionName": "openvino-dextr"}
21.07.20 11:44:06.726                     nuctl (I) Staging files and preparing base images
21.07.20 11:44:06.726                     nuctl (I) Building processor image {"imageName": "cvat/openvino.dextr:latest"}
21.07.20 11:44:06.726     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64"}
21.07.20 11:44:09.245     nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-365675208/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-c3r9lqavo2jvcpe0i3kg -f /tmp/nuclio-build-365675208/staging/Dockerfile.onbuild  --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler --build-arg NUCLIO_LABEL=1.5.16  .", "stderr": "unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n"}
21.07.20 11:44:09.246                     nuctl (W) Failed to create a function; setting the function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:118\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:225\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:188\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:49\nFailed to build processor image\n    /nuclio/pkg/processor/build/builder.go:250\nFailed to build processor image", "errCauses": [{"error": "Failed to build image artifacts", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:118\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:225\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:188\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:49\nFailed to build image artifacts", "errorCauses": [{"error": "Failed to copy objects from onbuild", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:118\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:225\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:188\nFailed to copy objects from onbuild", "errorCauses": [{"error": "Failed to build onbuild image", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:118\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:225\nFailed to build onbuild image", "errorCauses": [{"error": "Failed to build", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:118\nFailed to build", "errorCauses": [{"error": "stdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory\n", "errorCauses": [{"error": "exit status 1"}]}]}]}]}]}]}

Error - exit status 1
    /nuclio/pkg/cmdrunner/shellrunner.go:96

Call stack:
stdout:

stderr:
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-365675208: no such file or directory

    /nuclio/pkg/cmdrunner/shellrunner.go:96
Failed to build
    /nuclio/pkg/dockerclient/shell.go:118
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:225
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:188
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:49

Context

Your Environment

• Git hash commit (git log -1): 0dbe0a579f87a9e01198bfd494d2ff4b537de671
• Docker version docker version: 19.03.13, build cd8016b6bc
• Are you using Docker Swarm or Kubernetes? No
• Operating System and version: Tested on latest updated Ubuntu 20.04 and 18.04 LTS
• nuctl version: 1.5.16

CVAT built with:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml -f components/serverless/docker-compose.serverless.yml up -d --build

• Other diagnostic information / logs:
  Log from local machine

nuctl_error.log

[nmanovic]

@inecho , I also don't know what it can be. I redirected the question to nuclio github. Somebody reported similar issues. I see that nuclio cannot create a directory inside /tmp but I have no idea why.

[nmanovic]

@inecho , I believe we can do nothing with the problem. Need help to investigate the issue on your side. I will close the issue. I hope we will be able to simplify automatic annotation deployment in the future to address such cases.

[slverpla]

@inecho, running into the same issue, did you manage to solve it?

[VinceJnz]

I've not been able to get it working

[rongirl]

@inecho, running into the same issue, did you manage to solve it?

I have the same issue did you solve it?

[VinceJnz]

I've given up and I'm using a different application

[slverpla]

@rongirl, I am not sure how i solved this issue, but i think i went back to the latest stable version of the branch. When they last deployed the code. I think the branch is called release 2.4.3