FFTW 3.3.5: Combining MPI and Threads

In certain cases, it may be advantageous to combine MPI Chris@42: (distributed-memory) and threads (shared-memory) parallelization. Chris@42: FFTW supports this, with certain caveats. For example, if you have a Chris@42: cluster of 4-processor shared-memory nodes, you may want to use Chris@42: threads within the nodes and MPI between the nodes, instead of MPI for Chris@42: all parallelization. Chris@42:

In particular, it is possible to seamlessly combine the MPI FFTW Chris@42: routines with the multi-threaded FFTW routines (see Multi-threaded FFTW). However, some care must be taken in the initialization code, Chris@42: which should look something like this: Chris@42:

Chris@42:

int threads_ok;
Chris@42: 
Chris@42: int main(int argc, char **argv)
Chris@42: {
Chris@42:     int provided;
Chris@42:     MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
Chris@42:     threads_ok = provided >= MPI_THREAD_FUNNELED;
Chris@42: 
Chris@42:     if (threads_ok) threads_ok = fftw_init_threads();
Chris@42:     fftw_mpi_init();
Chris@42: 
Chris@42:     ...
Chris@42:     if (threads_ok) fftw_plan_with_nthreads(...);
Chris@42:     ...
Chris@42:     
Chris@42:     MPI_Finalize();
Chris@42: }
Chris@42:

First, note that instead of calling MPI_Init, you should call Chris@42: MPI_Init_threads, which is the initialization routine defined Chris@42: by the MPI-2 standard to indicate to MPI that your program will be Chris@42: multithreaded. We pass MPI_THREAD_FUNNELED, which indicates Chris@42: that we will only call MPI routines from the main thread. (FFTW will Chris@42: launch additional threads internally, but the extra threads will not Chris@42: call MPI code.) (You may also pass MPI_THREAD_SERIALIZED or Chris@42: MPI_THREAD_MULTIPLE, which requests additional multithreading Chris@42: support from the MPI implementation, but this is not required by Chris@42: FFTW.) The provided parameter returns what level of threads Chris@42: support is actually supported by your MPI implementation; this Chris@42: must be at least MPI_THREAD_FUNNELED if you want to call Chris@42: the FFTW threads routines, so we define a global variable Chris@42: threads_ok to record this. You should only call Chris@42: fftw_init_threads or fftw_plan_with_nthreads if Chris@42: threads_ok is true. For more information on thread safety in Chris@42: MPI, see the Chris@42: MPI and Chris@42: Threads section of the MPI-2 standard. Chris@42: Chris@42:

Second, we must call fftw_init_threads before Chris@42: fftw_mpi_init. This is critical for technical reasons having Chris@42: to do with how FFTW initializes its list of algorithms. Chris@42:

Then, if you call fftw_plan_with_nthreads(N), every MPI Chris@42: process will launch (up to) N threads to parallelize its transforms. Chris@42:

For example, in the hypothetical cluster of 4-processor nodes, you Chris@42: might wish to launch only a single MPI process per node, and then call Chris@42: fftw_plan_with_nthreads(4) on each process to use all Chris@42: processors in the nodes. Chris@42:

This may or may not be faster than simply using as many MPI processes Chris@42: as you have processors, however. On the one hand, using threads Chris@42: within a node eliminates the need for explicit message passing within Chris@42: the node. On the other hand, FFTW’s transpose routines are not Chris@42: multi-threaded, and this means that the communications that do take Chris@42: place will not benefit from parallelization within the node. Chris@42: Moreover, many MPI implementations already have optimizations to Chris@42: exploit shared memory when it is available, so adding the Chris@42: multithreaded FFTW on top of this may be superfluous. Chris@42: Chris@42:

6.11 Combining MPI and Threads