Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
deef8d0
Add runDocker mode to rqd
DiegoTavares Oct 16, 2024
e4fc361
Merge branch 'master' into rqd_containerized_frame
DiegoTavares Oct 16, 2024
7582d1a
[EXPERIMENT] Add runDocker mode to rqd (#1541)
DiegoTavares Oct 16, 2024
e56b145
Run rqd as root for docker mode
DiegoTavares Oct 17, 2024
4e365d4
[EXPERIMENT] Rqd containerized frame (#1546)
DiegoTavares Oct 17, 2024
a2283f4
Fix logging for deleteFrame
DiegoTavares Oct 18, 2024
90fce59
Update jobspec on dispatch test description
DiegoTavares Oct 18, 2024
8563f09
Allow multiple OSs on a dockerized rqd
DiegoTavares Oct 18, 2024
8a3da0b
Merge branch 'containerized_rqd' into rqd_multiple_os
DiegoTavares Oct 18, 2024
b117568
Update placeholder branch for containerized_rqd (#1550)
DiegoTavares Oct 18, 2024
0867bd1
Minor fixes
DiegoTavares Oct 21, 2024
3f62ded
Get the correct OS for procs
DiegoTavares Oct 22, 2024
c4547a0
Handle frame user on container
DiegoTavares Oct 24, 2024
ad0a3de
Simplify docker entrypoint cmd
DiegoTavares Oct 24, 2024
eb0962c
Stop logging exception for numby import
DiegoTavares Oct 25, 2024
1b637d1
Add unit tests for runOnDocker
DiegoTavares Oct 25, 2024
ebb4438
[cuegui] Fix TypeError in Comment viewer: Handle job object as iterab…
ramonfigueiredo Oct 17, 2024
98e5506
[cuegui] Add Rocky 9 log root in cuegui.yaml (#1543)
ramonfigueiredo Oct 17, 2024
fb3dcd6
[tests] Change tests to not use setup.py, but use the unittest module…
lithorus Oct 22, 2024
ebe966b
Rqd tests were not being executed
DiegoTavares Oct 26, 2024
be436e0
[rqd] Avoid changing dict in place during iteration (#1554)
DiegoTavares Oct 24, 2024
c8a9c2b
Add script for converting imports in grpc python modules and remove 2…
lithorus Oct 25, 2024
3b1d6b5
Fix pytest path
DiegoTavares Oct 26, 2024
0100ad6
Fix issue on rqmachine tests
DiegoTavares Oct 28, 2024
08a2625
Fix unit tests for rqcore and rqconstants
DiegoTavares Oct 28, 2024
8d7c0a2
Add runDocker mode to rqd
DiegoTavares Oct 16, 2024
e113df4
[EXPERIMENT] Rqd containerized frame (#1546)
DiegoTavares Oct 17, 2024
a292228
[EXPERIMENT] Rqd containerized frame (#1546)
DiegoTavares Oct 17, 2024
2933075
Add unit tests for runOnDocker
DiegoTavares Oct 25, 2024
f0ed725
Merge branch 'master' into rqd_multiple_os
DiegoTavares Oct 29, 2024
5345a52
Fix python lint
DiegoTavares Oct 29, 2024
80c1720
Fix lint
DiegoTavares Oct 29, 2024
21242b9
Fix lint issues
DiegoTavares Oct 29, 2024
88b8892
Fix migration
DiegoTavares Oct 29, 2024
8eba834
Remove undesired file
DiegoTavares Oct 29, 2024
2cf7f23
Make sure users are created with passwords
DiegoTavares Oct 29, 2024
73778ba
Merge branch 'AcademySoftwareFoundation:master' into rqd_multiple_os
DiegoTavares Oct 29, 2024
4c237ac
Merge branch 'containerized_rqd' into rqd_multiple_os
DiegoTavares Oct 30, 2024
2826826
Rqd multiple os (#1563)
DiegoTavares Oct 30, 2024
4ed1ef6
Fix typo on dispatchQuery
DiegoTavares Oct 30, 2024
a35d19a
Handle some possibly None variables
DiegoTavares Oct 31, 2024
69ab7e1
Merge branch 'master' into rqd_multiple_os
DiegoTavares Oct 31, 2024
050af9f
Merge branch 'containerized_rqd' into rqd_multiple_os
DiegoTavares Oct 31, 2024
f1623dc
Rqd multiple os (#1566)
DiegoTavares Oct 31, 2024
9074446
frame containers should run with network=host
DiegoTavares Oct 31, 2024
18ab932
Remove temporary password from the ENTRYPOINT log
DiegoTavares Nov 1, 2024
9a254de
Merge branch 'containerized_rqd' into rqd_multiple_os
DiegoTavares Nov 1, 2024
cb22fa5
Merge branch 'master' into rqd_multiple_os
DiegoTavares Nov 5, 2024
42632ab
Add runDocker mode to rqd
DiegoTavares Oct 16, 2024
0671f11
Update placeholder branch for containerized_rqd (#1550)
DiegoTavares Oct 18, 2024
dbe8e92
[cuebot] Move dispatcher memory properties to opencue.properties (#1570)
DiegoTavares Nov 5, 2024
cadd008
Fix merge error
DiegoTavares Nov 5, 2024
a2846ee
Merge branch 'master' into rqd_multiple_os
DiegoTavares Nov 5, 2024
bfbabb1
Address review comments
DiegoTavares Nov 6, 2024
bd593f0
Fix issues acquiring frame pid from container
DiegoTavares Nov 12, 2024
a87fc74
Improve logging for runDocker
DiegoTavares Nov 15, 2024
3061aec
Handle psutil.ZombieProcess
DiegoTavares Nov 15, 2024
6141a77
Merge branch 'master' into rqd_multiple_os
DiegoTavares Nov 15, 2024
70a3790
Fix merge conflicts
DiegoTavares Nov 15, 2024
60fc308
Fix unit tests and lint
DiegoTavares Nov 15, 2024
fb02866
Drop support for python libs on aswf_2022
DiegoTavares Nov 15, 2024
c5b697b
Fix lint
DiegoTavares Nov 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions .github/workflows/testing-pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,6 @@ on:
branches: [ master ]

jobs:
test_python_2022:
name: Run Python Unit Tests (CY2022)
runs-on: ubuntu-22.04
container: aswf/ci-opencue:2022
env:
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
steps:
- uses: actions/checkout@v3
- name: Run Python Tests
run: ci/run_python_tests.sh --no-gui

test_cuebot_2022:
name: Build Cuebot and Run Unit Tests (CY2022)
runs-on: ubuntu-22.04
Expand Down
2 changes: 1 addition & 1 deletion VERSION.in
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.0
1.1
1 change: 0 additions & 1 deletion ci/run_python_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ PYTHONPATH=pycue python -m unittest discover -s cueadmin/tests -t cueadmin -p "*
PYTHONPATH=pycue:pyoutline python -m unittest discover -s cuesubmit/tests -t cuesubmit -p "*.py"
python -m pytest rqd/tests


# Xvfb no longer supports Python 2.
if [[ "$python_version" =~ "Python 3" && ${args[0]} != "--no-gui" ]]; then
ci/run_gui_test.sh
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,8 @@ public class DispatchFrame extends FrameEntity implements FrameInterface {

// A comma separated list of services
public String services;

// The Operational System this frame is expected to run in
public String os;
}

10 changes: 9 additions & 1 deletion cuebot/src/main/java/com/imageworks/spcue/DispatchHost.java
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public class DispatchHost extends Entity
public long gpuMemory;
public long idleGpuMemory;
public String tags;
public String os;
private String os;

public boolean isNimby;
public boolean isLocalDispatch = false;
Expand Down Expand Up @@ -81,6 +81,14 @@ public String getFacilityId() {
return facilityId;
}

public String[] getOs() {
return this.os.split(",");
}

public void setOs(String os) {
this.os = os;
}

public boolean canHandleNegativeCoresRequest(int requestedCores) {
// Request is positive, no need to test further.
if (requestedCores > 0) {
Expand Down
8 changes: 5 additions & 3 deletions cuebot/src/main/java/com/imageworks/spcue/VirtualProc.java
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ public String getName() {
* @param frame
* @return
*/
public static final VirtualProc build(DispatchHost host, DispatchFrame frame, String... selfishServices) {
public static final VirtualProc build(DispatchHost host,
DispatchFrame frame,
String... selfishServices) {
VirtualProc proc = new VirtualProc();
proc.allocationId = host.getAllocationId();
proc.hostId = host.getHostId();
Expand All @@ -94,7 +96,7 @@ public static final VirtualProc build(DispatchHost host, DispatchFrame frame, St
proc.jobId = frame.getJobId();
proc.showId = frame.getShowId();
proc.facilityId = frame.getFacilityId();
proc.os = host.os;
proc.os = frame.os;

proc.hostName = host.getName();
proc.unbooked = false;
Expand Down Expand Up @@ -238,7 +240,7 @@ public static final VirtualProc build(DispatchHost host,
proc.jobId = frame.getJobId();
proc.showId = frame.getShowId();
proc.facilityId = frame.getFacilityId();
proc.os = host.os;
proc.os = frame.os;

proc.hostName = host.getName();
proc.unbooked = false;
Expand Down
Loading
Loading