FFTW 3.3.8: Combining MPI and Threads

In certain cases, it may be advantageous to combine MPI Chris@82: (distributed-memory) and threads (shared-memory) parallelization. Chris@82: FFTW supports this, with certain caveats. For example, if you have a Chris@82: cluster of 4-processor shared-memory nodes, you may want to use Chris@82: threads within the nodes and MPI between the nodes, instead of MPI for Chris@82: all parallelization. Chris@82:

In particular, it is possible to seamlessly combine the MPI FFTW Chris@82: routines with the multi-threaded FFTW routines (see Multi-threaded FFTW). However, some care must be taken in the initialization code, Chris@82: which should look something like this: Chris@82:

Chris@82:

int threads_ok;
Chris@82: 
Chris@82: int main(int argc, char **argv)
Chris@82: {
Chris@82:     int provided;
Chris@82:     MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
Chris@82:     threads_ok = provided >= MPI_THREAD_FUNNELED;
Chris@82: 
Chris@82:     if (threads_ok) threads_ok = fftw_init_threads();
Chris@82:     fftw_mpi_init();
Chris@82: 
Chris@82:     ...
Chris@82:     if (threads_ok) fftw_plan_with_nthreads(...);
Chris@82:     ...
Chris@82:     
Chris@82:     MPI_Finalize();
Chris@82: }
Chris@82:

First, note that instead of calling MPI_Init, you should call Chris@82: MPI_Init_threads, which is the initialization routine defined Chris@82: by the MPI-2 standard to indicate to MPI that your program will be Chris@82: multithreaded. We pass MPI_THREAD_FUNNELED, which indicates Chris@82: that we will only call MPI routines from the main thread. (FFTW will Chris@82: launch additional threads internally, but the extra threads will not Chris@82: call MPI code.) (You may also pass MPI_THREAD_SERIALIZED or Chris@82: MPI_THREAD_MULTIPLE, which requests additional multithreading Chris@82: support from the MPI implementation, but this is not required by Chris@82: FFTW.) The provided parameter returns what level of threads Chris@82: support is actually supported by your MPI implementation; this Chris@82: must be at least MPI_THREAD_FUNNELED if you want to call Chris@82: the FFTW threads routines, so we define a global variable Chris@82: threads_ok to record this. You should only call Chris@82: fftw_init_threads or fftw_plan_with_nthreads if Chris@82: threads_ok is true. For more information on thread safety in Chris@82: MPI, see the Chris@82: MPI and Chris@82: Threads section of the MPI-2 standard. Chris@82: Chris@82:

Second, we must call fftw_init_threads before Chris@82: fftw_mpi_init. This is critical for technical reasons having Chris@82: to do with how FFTW initializes its list of algorithms. Chris@82:

Then, if you call fftw_plan_with_nthreads(N), every MPI Chris@82: process will launch (up to) N threads to parallelize its transforms. Chris@82:

For example, in the hypothetical cluster of 4-processor nodes, you Chris@82: might wish to launch only a single MPI process per node, and then call Chris@82: fftw_plan_with_nthreads(4) on each process to use all Chris@82: processors in the nodes. Chris@82:

This may or may not be faster than simply using as many MPI processes Chris@82: as you have processors, however. On the one hand, using threads Chris@82: within a node eliminates the need for explicit message passing within Chris@82: the node. On the other hand, FFTW’s transpose routines are not Chris@82: multi-threaded, and this means that the communications that do take Chris@82: place will not benefit from parallelization within the node. Chris@82: Moreover, many MPI implementations already have optimizations to Chris@82: exploit shared memory when it is available, so adding the Chris@82: multithreaded FFTW on top of this may be superfluous. Chris@82: Chris@82:

6.11 Combining MPI and Threads