Controlling MPI Process Placement

The mpiexec command controls how the ranks of the processes are allocated to the nodes in the cluster. By default, mpiexec uses round-robin assignment of ranks to the nodes. This placement algorithm may not be the best choice for your application, particularly for clusters with symmetric multi-processor (SMP) nodes.

Suppose that the geometry is <#ranks> = 4 and <#nodes> = 2, where adjacent pairs of ranks are assigned to each node (for example, for 2-way SMP nodes). Enter the command:

type machines.Windows

cat ~/mpd.hosts

The results should be something like:

clusternode1

clusternode2

Since each node of the cluster is a 2-way SMP, and four processes are to be used for the application, the next experiment will distribute the four processes such that two of the processes will execute on clusternode1 and two processes will execute on clusternode2. For example, you might issue the following commands:

mpiexec -n 2 -host clusternode1 .\testf : -n 2 
 -host clusternode2 .\testf
mpiexec -n 2 -host clusternode1 .\testf90 : -n 
 2 -host clusternode2 .\testf90
mpiexec -n 2 -host clusternode1 .\testc : -n 2 
 -host clusternode2 .\testc
mpiexec -n 2 -host clusternode1 .\testcpp : -n 
 2 -host clusternode2 .\testcpp

mpiexec -n 2 -host clusternode1 ./testf : -n 2 
 -host clusternode2 ./testf
mpiexec -n 2 -host clusternode1 ./testf90 : -n 
 2 -host clusternode2 ./testf90
mpiexec -n 2 -host clusternode1 ./testc : -n 2 
 -host clusternode2 ./testc
mpiexec -n 2 -host clusternode1 ./testcpp : -n 
 2 -host clusternode2 ./testcpp

The following output should be produced for the executable testc:

Hello world: rank 0 of 4 running on clusternode1
Hello world: rank 1 of 4 running on clusternode1
Hello world: rank 2 of 4 running on clusternode2
Hello world: rank 3 of 4 running on clusternode2

In general, if there are i nodes in the cluster and each node is j-way SMP system, then the mpiexec command-line syntax for distributing the i by j processes amongst the i by j processors within the cluster is:

mpiexec -n j -host <nodename-1> 
.\mpi_example :  \
                     -n  j -host <nodename-2> 
.\mpi_example : \
                     -n  j -host <nodename-3> 
.\mpi_example : \
                                ...
                     -n  j -host <nodename-i> 
.\mpi_example

mpiexec -n j -host <nodename-1> 
./mpi_example :  \
                     -n  j -host <nodename-2> 
./mpi_example : \
                     -n  j -host <nodename-3> 
./mpi_example : \
                                ...
                     -n  j -host <nodename-i> 
./mpi_example