Ac kubernetes

[Acribbs]

No description provided.

[Acribbs]

@snsansom @IanSudbery @nickilott @david-cgat @jscaber (and all). This is a big change. I have refactored the code to introduce a new base class called the Executor that is inherited by Slurm, Torque, SGE, Kubernetes and local (it replaces make_runner()). The main aim was to implement kubernetes functonality which I couldnt do with the current framework. I introduced a new class function in base_executor.py that is inherited in executor.py (There is also a kubernets.py which I will move into executor.py before merging). I have tested it locally and on slurm on my end and it works, but if others could give it a go that would be great. It also defaults to LocalExecutor if you dont specify it to run locally, but now I think about it, I think im going to make it raise an error as that keeps it consistent with the current behaviour when it complains if DRMMA API isnt found if its ran for cluster config but is ran locally. Thinking of implementing a lot more functionalities in my free time (when i travel on trains and planes) so would be happy to accept suggestions on what to implement next.

[nickilott]

Thanks Adam for keeping going with cgat-core! I'm happy to test it out on existing pipelines and will let you know here how it goes

[Acribbs]

Planning on merging this soon

[nickilott]

Hey Adam, I just pulled the branch and tried to run a pipeline and got the following error:

File /gpfs3/well/powrie/users/utu691/devel/venv/Python-3.10.8-GCCcore-12.2.0/skylake/lib/python3.10/site-packages/cgatcore/pipeline/executors.py, line 69, in run
self.monitor_job_completion(job_id)
AttributeError: 'SlurmExecutor' object has no attribute monitor_job_completion

I haven'[t had a chance to look into the source of the error in any detail at all but wanted to flag before the merge...

[Acribbs]

Thanks for this! I have no idea why the tests are not picking this up. Will investigate further. May need some more tests to capure this