New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support parallel builds with GC3Pie (REVIEW) #1008

Merged

boegel merged 123 commits into easybuilders:develop from riccardomurri:gc3pie

Jul 9, 2015

Contributor

riccardomurri commented Aug 13, 2014

This is meant to be the implementation of #1006.

Basic functionality works. Configuration for the GC3Pie part is still hard-coded; I need your input as to how you would like this to be exposed in EB.

At the moment, it requires the "trunk" version of GC3Pie, soon to be released as GC3Pie 2.3.

Contributor

hpcugentbot commented Aug 13, 2014

Automatic reply from Jenkins: Can I test this?

riccardomurri mentioned this pull request

Support parallel builds with GC3Pie #1006

Closed

Member

boegel commented Aug 13, 2014

Jenkins: ok to test

Member

boegel commented Aug 13, 2014

@riccardomurri: the previous comment will trigger our continuous integration server to rerun the unit tests on every set of commits being pushed in, you can check for results using the markers next to the commit ids

stdweird reviewed

View reviewed changes

easybuild/main.py

@@ @@ -421,7 +421,8 @@ def main(testing_data=(None, None, None)): @@
                       command = "unset TMPDIR && cd %s && eb %%(spec)s %s" % (curdir, quoted_opts)
                       _log.info("Command template for jobs: %s" % command)
                       if not testing:
-                          jobs = build_easyconfigs_in_parallel(command, ordered_ecs)
+                          backend = options.job # either 'gc3pie' or 'pbs'

Contributor

stdweird Aug 15, 2014

use options.jobbackend

Collaborator

fgeorgatos commented Aug 18, 2014

Hi, just in case this becomes useful for other people following this thread:
radical-cybertools (former work in it was under the SAGA umbrella) as of now has support for multiple backends built-in, including torque, pbspro, slurm, sge, lsf, loadleveler & fork():
https://github.com/radical-cybertools/radical.pilot/blob/master/src/radical/pilot/agent/radical-pilot-agent-multicore.py

Member

boegel commented Aug 18, 2014

@riccardomurri: are you familiar with Radical-pilot, and how it's different from gc3pie?

Collaborator

fgeorgatos commented Aug 18, 2014

just to release the pressure from Riccardo, "pilot" jobs are jobs that in effect are pushed towards a node/core which then take the responsibility to pull the workload from some central service. They are very commonly found on grid platforms, as means to reduce failure complexity (if the pilot won't start, your job wouldn't have started either) and/or implement user-side job policies.

Member

boegel commented Aug 18, 2014

@fgeorgates: ok, thanks

Contributor Author

riccardomurri commented Aug 23, 2014

Hi all,

sorry for the late reply -- but we've had a very busy time due to a large hardware failure.

I'm on vacation starting tomorrow and will be back mid-September. I guess this PR can wait. Thanks for your comments and the patience so far!

Ciao,
R

Member

boegel commented Aug 23, 2014

@riccardomurri: no worries! I'd love to see this going in soon, but vacation is waaaay more important.
If I can find the time, I might pick this up and continue your effort myself.
I'll also be looking at gc3pie for a couple of other projects we're working on, it looks really promising!

boegel modified the milestone: v1.16

boegel force-pushed the develop branch from fe70342 to b2278f0 Compare

December 18, 2014 08:49


          tell GC3Pie engine to not rename output directory all the time

87a79ce

boegel mentioned this pull request

various fixes riccardomurri/easybuild-framework#19

Merged

riccardomurri and others added 3 commits

July 2, 2015 08:11


          Merge pull request #19 from boegel/gc3pie

c5302a6

various fixes


          fix remarks

a9a0309


          include comment on job_cnt

932e0e7

Contributor

hpcugentbot commented Jul 2, 2015

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1821/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1821/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel and others added 3 commits

July 2, 2015 14:16


          look into making GC3Pie not ignore errors (WIP)

e17593c


          set retrieve_overwrites to True in GC3Pie engine, fix unit test to ch…

380e459

…eck whether output dir is NOT renamed


          Merge pull request #20 from boegel/gc3pie

d75f6b1

make GC3Pie not ignore errors

Contributor

hpcugentbot commented Jul 2, 2015

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1827/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1827/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel and others added 3 commits

July 2, 2015 18:08


          bump minimal required SVN revision to 4255 (which includes fix for au…

2541c7b

…th=none)


          Merge pull request #21 from boegel/gc3pie

006bb6b

bump minimal required SVN revision to 4255 (which includes fix for auth=none)


          Merge branch 'develop' into gc3pie

c5739ad

Contributor

hpcugentbot commented Jul 2, 2015

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1828/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1828/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.


          Merge pull request #22 from boegel/gc3pie

38587f6

sync with develop

Contributor

hpcugentbot commented Jul 2, 2015

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1829/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1829/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel and others added 2 commits

July 6, 2015 11:10


          set resource_errors_are_fatal=True in create_engine call

3169a47


          Merge pull request #23 from boegel/gc3pie

f655ec2

set resource_errors_are_fatal=True in create_engine call

Contributor

hpcugentbot commented Jul 6, 2015

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1849/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1849/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel changed the title ~~Support parallel builds with GC3Pie (#1006, WIP)~~ Support parallel builds with GC3Pie (REVIEW)

boegel and others added 2 commits

July 8, 2015 21:23


          style fixes in job/backend.py and parallelbuild.py

1f487a1


          Merge pull request #24 from boegel/gc3pie

904d5b3

style fixes in job/backend.py and parallelbuild.py

Contributor

hpcugentbot commented Jul 8, 2015

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1865/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1865/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.


          Merge branch 'develop' into gc3pie

b3c59fc

boegel mentioned this pull request

sync with develop & resolve merge conflict riccardomurri/easybuild-framework#25

Merged


          Merge pull request #25 from boegel/gc3pie

c648f8e

sync with develop & resolve merge conflict

Contributor

hpcugentbot commented Jul 9, 2015

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1873/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1873/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

Member

boegel commented Jul 9, 2015

After a lot of work by both @riccardomurri and me on this, spread out across a long time (this effort started in Aug'14!), this is finally ready to go in.

We've tested this thoroughly, both on TORQUE and SLURM systems, and I've made sure that this doesn't affect the --regtest functionality I heavily rely upon for EB releases.

Documentation update is ready to go as well, see easybuilders/easybuild#127.

Thanks a lot @riccardomurri for working together on this with us, I expect that this will quickly become a much welcomed feature.

Merging to be included in EasyBuild v2.2!

boegel added a commit that referenced this pull request


          Merge pull request #1008 from gc3-uzh-ch/gc3pie

42ee7ae

Support parallel builds with GC3Pie (REVIEW)

boegel merged commit 42ee7ae into easybuilders:develop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet