[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549

DiegoTavares · 2024-10-18T23:38:56Z

Motivation

Running OpenCue In a multi operational system environment requires segregating the farm, which means hosts have to be assigned to one OS and cannot be shared between shows that have different OS requirements. This can be a challenge when sharing resources between shows is necessary.

Proposed solution

A new execution mode on rqd runDocker to live alongside runLinux, runWindows, and runDarwin (macOs). This mode will launch the frame command on a docker container based on the frame expected OS. With this, rqd is now able to run jobs from different OSs on the same host.

But to make this possible, a rqd host needs to advertise itself not with its own OS code (defined by SP_OS on rqd.conf), but with all the OSs of images it is capable of executing.

Configuration changes

The following sections were added to rqd.conf:

[docker.config]
# Setting this to True requires all the additional "docker.[]" sections to be filled
RUN_ON_DOCKER=True

# This section is only required if RUN_ON_DOCKER=True
# List of volume mounts following docker run's format, but replacing = with :
[docker.mounts]
TEMP=type:bind,source:/tmp,target:/tmp,bind-propagation:slave
NET=type:bind,source:/net,target:/net,bind-propagation:slave

# This section is only required if RUN_ON_DOCKER=True
#  - keys represent OSs this rqd is capable of executing jobs in
#  - values are docker image tags
[docker.images]
centos7=centos7.3:latest
rocky9=rocky9.3:latest

In this case, the rqd host would advertise itself with OS=centos7,rocky9, and the dispatch logic has been changed accordingly to account for dispatching frames to nodes that support multiple OSs.

When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Logging was added on the wrong scope, which led to a "Frame not found in cache" when a frame was actually found.

New spec is required to allow passing the layer's expected OS.

When rqd is running on docker mode, it can report multiple supported OSs. On rqd.conf, multiple images can be provided under [docker.images] and each image refers to a supported OS.

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

…ation#1550) Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Previously it was safe to use the host's OS when querying for procs, now the job OS needs to be used as a host can have multiple OSs.

To be able to run as the frame's owner, the entrypoint needs to ensure the user exists before running the frame's cmd.

Not having nimby installed is an expected event, not an exception.

…le (AcademySoftwareFoundation#1542) - Updated `viewComments` method in `MenuActions.py` to wrap single Job objects in a list. - This prevents `TypeError` when attempting to iterate over a non-iterable Job object.

…on#1543) - Add `rocky9` log root to `render_logs.root` in `cuegui.yaml`

… directly (AcademySoftwareFoundation#1547) **Summarize your change.** Have changed most tests to use `-m unittest discover` instead og `setup.py test` The old `setup.py test` doesn't work in newer versions of python since it has been deprecated

unittest was not reporting test failures and interruptions as expected, which caused us to be running with failed unit tests for a long time. This commit replaces unittest with pytest for rqd and fixes some of the relevant unit tests.

…oundation#1554) Deleting an item from the dict being iterated over on sanitizeFrames caused the error: "Dictionary changed size during iteration".

…to3 (AcademySoftwareFoundation#1557) **Link the Issue(s) this Pull Request is related to.** This is to fix AcademySoftwareFoundation#1555 **Summarize your change.** Replaces 2to3 with a simple script that adds "from ." in front of pb2 imports. This is done to support newer versions of python where 2to3 has been removed.

Since AcademySoftwareFoundation#1308 rqd stopped supporting stats files containing whitespaces and parenthesis.

When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

DiegoTavares · 2024-10-28T23:59:52Z

This change has been rebased from #1560 to allow running unit tests on rqd.

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Update temporary sync branch --------- Signed-off-by: Diego Tavares <dtavares@imageworks.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com> Co-authored-by: Jimmy Christensen <Lithorus@gmail.com>

For services as SMTP and others that require direct access to a port, running with network HOST gives frames a similar access to network as they had when running outside of a container

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.

…ation#1550) Signed-off-by: Diego Tavares <dtavares@imageworks.com>

…demySoftwareFoundation#1570) Memory properties constantly need to be tuned according to farm requirements, which makes it a good candidate for becoming a property instead of a hardcoded constant.

rqd/tests/rqnimby_test.py

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

ramonfigueiredo

LGFM

Approved with minor changes.

Thanks!

cuebot/src/main/java/com/imageworks/spcue/dao/postgres/DispatcherDaoJdbc.java

cuebot/src/main/java/com/imageworks/spcue/dispatcher/HostReportHandler.java

rqd/rqd/rqconstants.py

rqd/rqd/rqcore.py

See AcademySoftwareFoundation/OpenCue#1549 for more details

lithorus · 2024-11-12T20:41:42Z

Any idea on when this will get merged? I have a PR coming with the loki support which will have several merge conflicts with this branch PR :)

Using the container logs to get the frameId is not reliable. When the container fails quick docker doesn't stream the logs, so a new strategy using container.top() was implemented failing back to the log solution if needed be.

Besides that, also add escaping for " on the frame command being sent to docker.

calling psutil's function cmdline raises the ZombieProcess, which wasn't been caught and caused an interuptino on the rssUpdate loop.

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

DiegoTavares · 2024-11-15T17:32:28Z

Any idea on when this will get merged? I have a PR coming with the loki support which will have several merge conflicts with this branch PR :)

Today

Docker library is incompatible with OpenSSL<1.1.1+(2017)

DiegoTavares and others added 29 commits October 16, 2024 15:55

Add runDocker mode to rqd

deef8d0

When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.

Merge branch 'master' into rqd_containerized_frame

e4fc361

[EXPERIMENT] Add runDocker mode to rqd (AcademySoftwareFoundation#1541)

7582d1a

When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.

Run rqd as root for docker mode

e56b145

[EXPERIMENT] Rqd containerized frame (AcademySoftwareFoundation#1546)

4e365d4

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Fix logging for deleteFrame

a2283f4

Logging was added on the wrong scope, which led to a "Frame not found in cache" when a frame was actually found.

Update jobspec on dispatch test description

90fce59

New spec is required to allow passing the layer's expected OS.

Allow multiple OSs on a dockerized rqd

8563f09

When rqd is running on docker mode, it can report multiple supported OSs. On rqd.conf, multiple images can be provided under [docker.images] and each image refers to a supported OS.

Merge branch 'containerized_rqd' into rqd_multiple_os

8a3da0b

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Update placeholder branch for containerized_rqd (AcademySoftwareFound…

b117568

…ation#1550) Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Minor fixes

0867bd1

Get the correct OS for procs

3f62ded

Previously it was safe to use the host's OS when querying for procs, now the job OS needs to be used as a host can have multiple OSs.

Handle frame user on container

c4547a0

To be able to run as the frame's owner, the entrypoint needs to ensure the user exists before running the frame's cmd.

Simplify docker entrypoint cmd

ad0a3de

Stop logging exception for numby import

eb0962c

Not having nimby installed is an expected event, not an exception.

Add unit tests for runOnDocker

1b637d1

[cuegui] Add Rocky 9 log root in cuegui.yaml (AcademySoftwareFoundati…

98e5506

…on#1543) - Add `rocky9` log root to `render_logs.root` in `cuegui.yaml`

Rqd tests were not being executed

ebe966b

unittest was not reporting test failures and interruptions as expected, which caused us to be running with failed unit tests for a long time. This commit replaces unittest with pytest for rqd and fixes some of the relevant unit tests.

[rqd] Avoid changing dict in place during iteration (AcademySoftwareF…

be436e0

…oundation#1554) Deleting an item from the dict being iterated over on sanitizeFrames caused the error: "Dictionary changed size during iteration".

Fix pytest path

3b1d6b5

Fix issue on rqmachine tests

0100ad6

Since AcademySoftwareFoundation#1308 rqd stopped supporting stats files containing whitespaces and parenthesis.

Fix unit tests for rqcore and rqconstants

08a2625

Add runDocker mode to rqd

8d7c0a2

When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.

[EXPERIMENT] Rqd containerized frame (AcademySoftwareFoundation#1546)

e113df4

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

[EXPERIMENT] Rqd containerized frame (AcademySoftwareFoundation#1546)

a292228

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Add unit tests for runOnDocker

2933075

DiegoTavares and others added 13 commits October 30, 2024 14:14

Fix typo on dispatchQuery

4ed1ef6

Handle some possibly None variables

a35d19a

Merge branch 'master' into rqd_multiple_os

69ab7e1

Merge branch 'containerized_rqd' into rqd_multiple_os

050af9f

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Rqd multiple os (AcademySoftwareFoundation#1566)

f1623dc

Update temporary sync branch --------- Signed-off-by: Diego Tavares <dtavares@imageworks.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com> Co-authored-by: Jimmy Christensen <Lithorus@gmail.com>

frame containers should run with network=host

9074446

For services as SMTP and others that require direct access to a port, running with network HOST gives frames a similar access to network as they had when running outside of a container

Remove temporary password from the ENTRYPOINT log

18ab932

Merge branch 'containerized_rqd' into rqd_multiple_os

9a254de

Merge branch 'master' into rqd_multiple_os

cb22fa5

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

Add runDocker mode to rqd

42632ab

When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.

Update placeholder branch for containerized_rqd (AcademySoftwareFound…

0671f11

…ation#1550) Signed-off-by: Diego Tavares <dtavares@imageworks.com>

[cuebot] Move dispatcher memory properties to opencue.properties (Aca…

dbe8e92

…demySoftwareFoundation#1570) Memory properties constantly need to be tuned according to farm requirements, which makes it a good candidate for becoming a property instead of a hardcoded constant.

Fix merge error

cadd008

ramonfigueiredo reviewed Nov 5, 2024

View reviewed changes

rqd/tests/rqnimby_test.py Outdated Show resolved Hide resolved

Merge branch 'master' into rqd_multiple_os

a2846ee

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

ramonfigueiredo approved these changes Nov 5, 2024

View reviewed changes

Address review comments

bfbabb1

DiegoTavares added a commit to AcademySoftwareFoundation/opencue.io that referenced this pull request Nov 7, 2024

Add docs for containerize frames feature

243a2e0

See AcademySoftwareFoundation/OpenCue#1549 for more details

DiegoTavares mentioned this pull request Nov 7, 2024

Add docs for containerize frames feature AcademySoftwareFoundation/opencue.io#302

Open

DiegoTavares added 4 commits November 12, 2024 15:18

Fix issues acquiring frame pid from container

bd593f0

Using the container logs to get the frameId is not reliable. When the container fails quick docker doesn't stream the logs, so a new strategy using container.top() was implemented failing back to the log solution if needed be.

Improve logging for runDocker

a87fc74

Besides that, also add escaping for " on the frame command being sent to docker.

Handle psutil.ZombieProcess

3061aec

calling psutil's function cmdline raises the ZombieProcess, which wasn't been caught and caused an interuptino on the rssUpdate loop.

Merge branch 'master' into rqd_multiple_os

6141a77

Signed-off-by: Diego Tavares <dtavares@imageworks.com>

DiegoTavares added 4 commits November 15, 2024 09:33

Fix merge conflicts

70a3790

Fix unit tests and lint

60fc308

Drop support for python libs on aswf_2022

fb02866

Docker library is incompatible with OpenSSL<1.1.1+(2017)

Fix lint

c5b697b

DiegoTavares merged commit 291b694 into AcademySoftwareFoundation:master Nov 15, 2024
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549

[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549

DiegoTavares commented Oct 18, 2024 •

edited by ramonfigueiredo

Loading

DiegoTavares commented Oct 28, 2024

ramonfigueiredo left a comment

lithorus commented Nov 12, 2024

DiegoTavares commented Nov 15, 2024

[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549

[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549

Conversation

DiegoTavares commented Oct 18, 2024 • edited by ramonfigueiredo Loading

Motivation

Proposed solution

Configuration changes

DiegoTavares commented Oct 28, 2024

ramonfigueiredo left a comment

Choose a reason for hiding this comment

lithorus commented Nov 12, 2024

DiegoTavares commented Nov 15, 2024

DiegoTavares commented Oct 18, 2024 •

edited by ramonfigueiredo

Loading