Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: update parallel-computing for ClusterManager #4014

Merged
merged 1 commit into from
Aug 11, 2013
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions doc/manual/parallel-computing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -589,3 +589,38 @@ required, since the threads are scheduled cooperatively and not
preemptively. This means context switches only occur at well-defined
points: in this case, when ``remotecall_fetch`` is called.


ClusterManagers
---------------

Julia worker processes can also be spawned on arbitrary machines,
enabling Julia's natural parallelism to function quite transparently
in a cluster environment. The ``ClusterManager`` interface provides a
way to specify a means to launch and manage worker processes, for
example::

immutable SSHManager <: ClusterManager
launch::Function
manage::Function
machines::AbstractVector

SSHManager(; machines=[]) = new(launch_ssh_workers, manage_ssh_workers, machines)
end

function launch_ssh_workers(cman::SSHManager, np::Integer, config::Dict)
...
end

function manage_ssh_workers(id::Integer, config::Dict, op::Symbol)
...
end

where ``launch_ssh_workers`` is responsible for instantiating new
Julia processes and ``manage_ssh_workers`` provides a means to manage
those processes, e.g. for sending interrupt signals. New processes can
then be added at runtime using ``addprocs``::

addprocs(5, cman=LocalManager())

which specifies a number of processes to add and a ``ClusterManager`` to
use for launching those processes.