Chris@10: Next: FFTW MPI Reference, Chris@10: Previous: FFTW MPI Performance Tips, Chris@10: Up: Distributed-memory FFTW with MPI Chris@10:
Chris@10: In certain cases, it may be advantageous to combine MPI Chris@10: (distributed-memory) and threads (shared-memory) parallelization. Chris@10: FFTW supports this, with certain caveats. For example, if you have a Chris@10: cluster of 4-processor shared-memory nodes, you may want to use Chris@10: threads within the nodes and MPI between the nodes, instead of MPI for Chris@10: all parallelization. Chris@10: Chris@10:
In particular, it is possible to seamlessly combine the MPI FFTW Chris@10: routines with the multi-threaded FFTW routines (see Multi-threaded FFTW). However, some care must be taken in the initialization code, Chris@10: which should look something like this: Chris@10: Chris@10:
int threads_ok;
Chris@10:
Chris@10: int main(int argc, char **argv)
Chris@10: {
Chris@10: int provided;
Chris@10: MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
Chris@10: threads_ok = provided >= MPI_THREAD_FUNNELED;
Chris@10:
Chris@10: if (threads_ok) threads_ok = fftw_init_threads();
Chris@10: fftw_mpi_init();
Chris@10:
Chris@10: ...
Chris@10: if (threads_ok) fftw_plan_with_nthreads(...);
Chris@10: ...
Chris@10:
Chris@10: MPI_Finalize();
Chris@10: }
Chris@10:
Chris@10:
Chris@10: First, note that instead of calling MPI_Init, you should call
Chris@10: MPI_Init_threads, which is the initialization routine defined
Chris@10: by the MPI-2 standard to indicate to MPI that your program will be
Chris@10: multithreaded. We pass MPI_THREAD_FUNNELED, which indicates
Chris@10: that we will only call MPI routines from the main thread. (FFTW will
Chris@10: launch additional threads internally, but the extra threads will not
Chris@10: call MPI code.) (You may also pass MPI_THREAD_SERIALIZED or
Chris@10: MPI_THREAD_MULTIPLE, which requests additional multithreading
Chris@10: support from the MPI implementation, but this is not required by
Chris@10: FFTW.) The provided parameter returns what level of threads
Chris@10: support is actually supported by your MPI implementation; this
Chris@10: must be at least MPI_THREAD_FUNNELED if you want to call
Chris@10: the FFTW threads routines, so we define a global variable
Chris@10: threads_ok to record this. You should only call
Chris@10: fftw_init_threads or fftw_plan_with_nthreads if
Chris@10: threads_ok is true. For more information on thread safety in
Chris@10: MPI, see the
Chris@10: MPI and Threads section of the MPI-2 standard.
Chris@10:
Chris@10:
Chris@10:
Second, we must call fftw_init_threads before
Chris@10: fftw_mpi_init. This is critical for technical reasons having
Chris@10: to do with how FFTW initializes its list of algorithms.
Chris@10:
Chris@10:
Then, if you call fftw_plan_with_nthreads(N), every MPI
Chris@10: process will launch (up to) N threads to parallelize its transforms.
Chris@10:
Chris@10:
For example, in the hypothetical cluster of 4-processor nodes, you
Chris@10: might wish to launch only a single MPI process per node, and then call
Chris@10: fftw_plan_with_nthreads(4) on each process to use all
Chris@10: processors in the nodes.
Chris@10:
Chris@10:
This may or may not be faster than simply using as many MPI processes Chris@10: as you have processors, however. On the one hand, using threads Chris@10: within a node eliminates the need for explicit message passing within Chris@10: the node. On the other hand, FFTW's transpose routines are not Chris@10: multi-threaded, and this means that the communications that do take Chris@10: place will not benefit from parallelization within the node. Chris@10: Moreover, many MPI implementations already have optimizations to Chris@10: exploit shared memory when it is available, so adding the Chris@10: multithreaded FFTW on top of this may be superfluous. Chris@10: Chris@10: Chris@10: Chris@10: Chris@10: