run cactus on the grid_engine cluster #5022
Comments
|
@minglibio Thanks for reporting, I don't think the Regarding running cactus on specific queues, Cactus uses Toil to run it's entire workflow, meaning any Toil argument should apply to the cactus run itself. Those two environment variables should be the only ones you need to set. Although I'm unsure if passing multiple queues to the batch system is supported (this is likely dependent on the batch system and not Toil). If |
|
➤ Adam Novak commented: I think part of fixing this might be trawling the GridEngineBatchSystem for any required methods that aren’t implemented. We don’t use actual Grid Engine itself in CI so it’s possible we added a NotInplementedError abstract method to AbstractGridEngineBatchSystem and forgot the implementation here. |
|
Hey! The error is thrown basically immediately. The summary of the toil requirements also seem fine, so I'm not sure why it would have depricated stuff. Best, |
|
I'm not sure if this is helpful or relevant but I tried re-installing everything with cactus 2.9 in an otherwise empty environment. The toil requirements text file is this While I didn't get any notes or warnings, I did see this line in the installation: Could this be leading to an issue with running toil on SGE? Best, |
|
I think the NotImplementedError may be a bit misleading. The function The backports message should be fine when running Python 3.9 and above, as the #5061 should fix this logging error though. You can install Toil from source at that branch by running or You may need to uninstall toil first with should work. @DustinSokolowski @minglibio We don't have a SGE cluster here to test on, so please do tell us if the fix works. |
|
Hey! Thank you for the quick response. So far so good! 
It also looks like cactus was able to run with the "singleComputer" option when submitting jobs to the SGE, which is encouraging. I will open up a new issue if I run into issues with cactus using "grid_engine". We're assembling a number of species in the same family tree and are hoping to do an MSA using cactus on our SGE cluster, so it will be nice to get cactus working in both modes. best, |
|
Hey @stxue1 Thanks for your help. After installed the new toil, the NotImplementedError was successfully fixed. But there are still some errors to prevent me from running Attached is the log. |
|
Hey @minglibio I was looking through the error file and I noticed that your sge is fully rejecting the first job
Is there a possibility that there is an incompatibility with your queue commands and toils? For example for us we need to assign a project and a job name or else the cluster rejects us outright (albiet it tells us why). Do you know if you have to set a min hmem or something like that? |
|
I tested it, and it can run a job without any parameters. In this way, the job name will be assigned as the shell file name. I ran the inner example of cactus (only a few species and a few parts of chromosome 5) and I don't expect it will generate so many jobs... I never submit 10k jobs one time, but 1k should be fine in our I noticed that in your last run, you used Best, |
|
Hey! Yeah I was sort of trying everything RE: cactus version. I'm not sure it made a difference. I'm not sure I have a great answer for your question. I haven't made it to the end of a pipeline on grid_engine mode (though I now have on singleMachine). Here's a screenshot of the current log. It's able to run a lot of jobs and some jobs also fail. I think this is the expected behaviour (and toil retries). |
|
The line from the log Since the |
|
A guess that _toil_worker is being executed as a shell script rather
that a python program.
what do the first 10 lines of the file look like on your system?
stxue1 ***@***.***> writes:
… The line from the log `import: unable to open X server `' @ error/import.c/ImportImageCommand/349.` kind of sounds like it's trying to run a [python script as a bash script](https://stackoverflow.com/questions/19507096/python-error-import-unable-to-open-x-server).
Since the `_toil_worker` seems to be invoked on the cactus side as some sort of binary/pointer, this could be more of a cactus issue. @glennhickey @diekhans Does the log look more like a cactus related problem to you?
```
=========>
import: unable to open X server `' @ error/import.c/ImportImageCommand/349.
import: unable to open X server `' @ error/import.c/ImportImageCommand/349.
/data/scc3/ming.li/software/cactus-bin-v2.8.4/venv-cactus-v2.8.4/bin/_toil_worker: line 5: from: command not found
/data/scc3/ming.li/software/cactus-bin-v2.8.4/venv-cactus-v2.8.4/bin/_toil_worker: _toil_worker: line 7: syntax error near unexpected token `('
/data/scc3/ming.li/software/cactus-bin-v2.8.4/venv-cactus-v2.8.4/bin/_toil_worker: _toil_worker: line 7: ` sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'
<=========
```
--
Reply to this email directly or view it on GitHub:
#5022 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
|
You mean the Here is the file: |
|
Reopening as the issue itself isn't fully resolved.
Looks like it is being executed as a bash/shell script instead of python. The toil worker file matches the logged error.
|
|
Should I open an issue in cactus to resolve this one? |
At least I don't think there's much we can do on the Toil side for this specific issue. Though my hunch is that this is more of a configuration issue than a toil/cactus issue. The bash shebang at the top of the script should ensure that the file is ran by the referenced executable when ran as a bash script. Does #!/data/scc3/ming.li/software/cactus-bin-v2.8.4/venv-cactus-v2.8.4/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
print("Hello world")and running |
|
I tested, and all things went well. I also tested the What I will do is try to figure it out with our cluster manager if it is bc the setting of our cluster. I will let you know if we can solve this problem. |
It's likely also worth testing if the |
|
It is bc the system setting of the cluster, all queues on our cluster are set to posix_compliant as shell_start_mode which ignores the #! line. add the -S parameter resolved this issue. |
Hi,
I am trying to run
cactus v2.8.4on a grid_engine cluster with the following command:I am keeping to get the error
NotImplementedError.Should I do anything to avoid this error?
Other question:
In this cluster, we have several different queues. I want to run cactus on some specific queues bc they are faster. Which parameter should I add to achieve this need? Should I add the following environment variables?
Best,
Ming
┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1617
The text was updated successfully, but these errors were encountered: