Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sub tasks with the same label blocking eachother #12

Open
isthisthat opened this issue Aug 25, 2015 · 1 comment
Open

sub tasks with the same label blocking eachother #12

isthisthat opened this issue Aug 25, 2015 · 1 comment

Comments

@isthisthat
Copy link

Hello,
I was trying to debug this for a very long time.. It seems that if you have sub-tasks with the same label they will not run in parallel. For example if you have:

for i in [1,2,3]:
    self.addWorkflowTask("task-%d" % i, RunFoo())

class RunFoo(WorkflowRunner):
    def workflow(self):
        self.addTask("foo", cmd)
        self.addTask("bar", cmd)

Only one "foo" and "bar" will run in parallel, even though their labels are "task-1+foo", "task-1+bar", "task-2+foo", "task-2+bar", "task-3+foo", "task-3+bar"

A quick workaround is this:

for i in [1,2,3]:
    self.addWorkflowTask("task-%d" % i, RunFoo(pid))

class RunFoo(WorkflowRunner):
    def __init__(self, pid):
        self.pid = pid

    def workflow(self):
        self.addTask("foo-%d" % self.pid, cmd)
        self.addTask("bar-%d" % self.pid, cmd)

Which then creates globally unique sub-tasks and solves the problem.
Wondering if this is a bug.

@ctsa
Copy link
Contributor

ctsa commented Aug 26, 2015

Thanks for the report -- I won't be able to look at this for a bit but if confirmed it is a bug. If you'd like to see this addressed sooner, it would help if you could create a fork/branch with a new demo item showing the behavior -- perhaps two simple long-term sleep tasks that run serially when the pyflow job is given two cores and sufficient memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants