comparison src/fftw-3.3.5/doc/mpi.texi @ 127:7867fa7e1b6b

Current fftw source

+@node Distributed-memory FFTW with MPI, Calling FFTW from Modern Fortran, Multi-threaded FFTW, Top
+@chapter Distributed-memory FFTW with MPI
+@cindex MPI
+
+@cindex parallel transform
+In this chapter we document the parallel FFTW routines for parallel
+systems supporting the MPI message-passing interface.  Unlike the
+shared-memory threads described in the previous chapter, MPI allows
+you to use @emph{distributed-memory} parallelism, where each CPU has
+its own separate memory, and which can scale up to clusters of many
+thousands of processors.  This capability comes at a price, however:
+each process only stores a @emph{portion} of the data to be
+transformed, which means that the data structures and
+programming-interface are quite different from the serial or threads
+versions of FFTW.
+@cindex data distribution
+
+
+Distributed-memory parallelism is especially useful when you are
+transforming arrays so large that they do not fit into the memory of a
+single processor.  The storage per-process required by FFTW's MPI
+routines is proportional to the total array size divided by the number
+of processes.  Conversely, distributed-memory parallelism can easily
+pose an unacceptably high communications overhead for small problems;
+the threshold problem size for which parallelism becomes advantageous
+will depend on the precise problem you are interested in, your
+hardware, and your MPI implementation.
+
+A note on terminology: in MPI, you divide the data among a set of
+``processes'' which each run in their own memory address space.
+Generally, each process runs on a different physical processor, but
+this is not required.  A set of processes in MPI is described by an
+opaque data structure called a ``communicator,'' the most common of
+which is the predefined communicator @code{MPI_COMM_WORLD} which
+refers to @emph{all} processes.  For more information on these and
+other concepts common to all MPI programs, we refer the reader to the
+documentation at @uref{http://www.mcs.anl.gov/research/projects/mpi/, the MPI home
+page}.
+@cindex MPI communicator
+@ctindex MPI_COMM_WORLD
+
+
+We assume in this chapter that the reader is familiar with the usage
+of the serial (uniprocessor) FFTW, and focus only on the concepts new
+to the MPI interface.
+
+@menu
+* FFTW MPI Installation::
+* Linking and Initializing MPI FFTW::
+* 2d MPI example::
+* MPI Data Distribution::
+* Multi-dimensional MPI DFTs of Real Data::
+* Other Multi-dimensional Real-data MPI Transforms::
+* FFTW MPI Transposes::
+* FFTW MPI Wisdom::
+* Avoiding MPI Deadlocks::
+* FFTW MPI Performance Tips::
+* Combining MPI and Threads::
+* FFTW MPI Reference::
+* FFTW MPI Fortran Interface::
+@end menu
+
+@c ------------------------------------------------------------
+@node FFTW MPI Installation, Linking and Initializing MPI FFTW, Distributed-memory FFTW with MPI, Distributed-memory FFTW with MPI
+@section FFTW MPI Installation
+
+All of the FFTW MPI code is located in the @code{mpi} subdirectory of
+the FFTW package.  On Unix systems, the FFTW MPI libraries and header
+files are automatically configured, compiled, and installed along with
+the uniprocessor FFTW libraries simply by including
+@code{--enable-mpi} in the flags to the @code{configure} script
+(@pxref{Installation on Unix}).
+@fpindex configure
+
+
+Any implementation of the MPI standard, version 1 or later, should
+work with FFTW.  The @code{configure} script will attempt to
+automatically detect how to compile and link code using your MPI
+implementation.  In some cases, especially if you have multiple
+different MPI implementations installed or have an unusual MPI
+software package, you may need to provide this information explicitly.
+
+Most commonly, one compiles MPI code by invoking a special compiler
+command, typically @code{mpicc} for C code.  The @code{configure}
+script knows the most common names for this command, but you can
+specify the MPI compilation command explicitly by setting the
+@code{MPICC} variable, as in @samp{./configure MPICC=mpicc ...}.
+@fpindex mpicc
+
+
+If, instead of a special compiler command, you need to link a certain
+library, you can specify the link command via the @code{MPILIBS}
+variable, as in @samp{./configure MPILIBS=-lmpi ...}.  Note that if
+your MPI library is installed in a non-standard location (one the
+compiler does not know about by default), you may also have to specify
+the location of the library and header files via @code{LDFLAGS} and
+@code{CPPFLAGS} variables, respectively, as in @samp{./configure
+LDFLAGS=-L/path/to/mpi/libs CPPFLAGS=-I/path/to/mpi/include ...}.
+
+@c ------------------------------------------------------------
+@node Linking and Initializing MPI FFTW, 2d MPI example, FFTW MPI Installation, Distributed-memory FFTW with MPI
+@section Linking and Initializing MPI FFTW
+
+Programs using the MPI FFTW routines should be linked with
+@code{-lfftw3_mpi -lfftw3 -lm} on Unix in double precision,
+@code{-lfftw3f_mpi -lfftw3f -lm} in single precision, and so on
+(@pxref{Precision}). You will also need to link with whatever library
+is responsible for MPI on your system; in most MPI implementations,
+there is a special compiler alias named @code{mpicc} to compile and
+link MPI code.
+@fpindex mpicc
+@cindex linking on Unix
+@cindex precision
+
+
+@findex fftw_init_threads
+Before calling any FFTW routines except possibly
+@code{fftw_init_threads} (@pxref{Combining MPI and Threads}), but after calling
+@code{MPI_Init}, you should call the function:
+
+@example
+void fftw_mpi_init(void);
+@end example
+@findex fftw_mpi_init
+
+If, at the end of your program, you want to get rid of all memory and
+other resources allocated internally by FFTW, for both the serial and
+MPI routines, you can call:
+
+@example
+void fftw_mpi_cleanup(void);
+@end example
+@findex fftw_mpi_cleanup
+
+which is much like the @code{fftw_cleanup()} function except that it
+also gets rid of FFTW's MPI-related data.  You must @emph{not} execute
+any previously created plans after calling this function.
+
+@c ------------------------------------------------------------
+@node 2d MPI example, MPI Data Distribution, Linking and Initializing MPI FFTW, Distributed-memory FFTW with MPI
+@section 2d MPI example
+
+Before we document the FFTW MPI interface in detail, we begin with a
+simple example outlining how one would perform a two-dimensional
+@code{N0} by @code{N1} complex DFT. 
+
+@example
+#include <fftw3-mpi.h>
+
+int main(int argc, char **argv)
+@{
+    const ptrdiff_t N0 = ..., N1 = ...;
+    fftw_plan plan;
+    fftw_complex *data;
+    ptrdiff_t alloc_local, local_n0, local_0_start, i, j;
+
+    MPI_Init(&argc, &argv);
+    fftw_mpi_init();
+
+    /* @r{get local data size and allocate} */
+    alloc_local = fftw_mpi_local_size_2d(N0, N1, MPI_COMM_WORLD,
+                                         &local_n0, &local_0_start);
+    data = fftw_alloc_complex(alloc_local);
+
+    /* @r{create plan for in-place forward DFT} */
+    plan = fftw_mpi_plan_dft_2d(N0, N1, data, data, MPI_COMM_WORLD,
+                                FFTW_FORWARD, FFTW_ESTIMATE);    
+
+    /* @r{initialize data to some function} my_function(x,y) */
+    for (i = 0; i < local_n0; ++i) for (j = 0; j < N1; ++j)
+       data[i*N1 + j] = my_function(local_0_start + i, j);
+
+    /* @r{compute transforms, in-place, as many times as desired} */
+    fftw_execute(plan);
+
+    fftw_destroy_plan(plan);
+
+    MPI_Finalize();
+@}
+@end example
+
+As can be seen above, the MPI interface follows the same basic style
+of allocate/plan/execute/destroy as the serial FFTW routines.  All of
+the MPI-specific routines are prefixed with @samp{fftw_mpi_} instead
+of @samp{fftw_}.  There are a few important differences, however:
+
+First, we must call @code{fftw_mpi_init()} after calling
+@code{MPI_Init} (required in all MPI programs) and before calling any
+other @samp{fftw_mpi_} routine.
+@findex MPI_Init
+@findex fftw_mpi_init
+
+
+Second, when we create the plan with @code{fftw_mpi_plan_dft_2d},
+analogous to @code{fftw_plan_dft_2d}, we pass an additional argument:
+the communicator, indicating which processes will participate in the
+transform (here @code{MPI_COMM_WORLD}, indicating all processes).
+Whenever you create, execute, or destroy a plan for an MPI transform,
+you must call the corresponding FFTW routine on @emph{all} processes
+in the communicator for that transform.  (That is, these are
+@emph{collective} calls.)  Note that the plan for the MPI transform
+uses the standard @code{fftw_execute} and @code{fftw_destroy} routines
+(on the other hand, there are MPI-specific new-array execute functions
+documented below).
+@cindex collective function
+@findex fftw_mpi_plan_dft_2d
+@ctindex MPI_COMM_WORLD
+
+
+Third, all of the FFTW MPI routines take @code{ptrdiff_t} arguments
+instead of @code{int} as for the serial FFTW.  @code{ptrdiff_t} is a
+standard C integer type which is (at least) 32 bits wide on a 32-bit
+machine and 64 bits wide on a 64-bit machine.  This is to make it easy
+to specify very large parallel transforms on a 64-bit machine.  (You
+can specify 64-bit transform sizes in the serial FFTW, too, but only
+by using the @samp{guru64} planner interface.  @xref{64-bit Guru
+Interface}.)
+@tindex ptrdiff_t
+@cindex 64-bit architecture
+
+
+Fourth, and most importantly, you don't allocate the entire
+two-dimensional array on each process.  Instead, you call
+@code{fftw_mpi_local_size_2d} to find out what @emph{portion} of the
+array resides on each processor, and how much space to allocate.
+Here, the portion of the array on each process is a @code{local_n0} by
+@code{N1} slice of the total array, starting at index
+@code{local_0_start}.  The total number of @code{fftw_complex} numbers
+to allocate is given by the @code{alloc_local} return value, which
+@emph{may} be greater than @code{local_n0 * N1} (in case some
+intermediate calculations require additional storage).  The data
+distribution in FFTW's MPI interface is described in more detail by
+the next section.
+@findex fftw_mpi_local_size_2d
+@cindex data distribution
+
+
+Given the portion of the array that resides on the local process, it
+is straightforward to initialize the data (here to a function
+@code{myfunction}) and otherwise manipulate it.  Of course, at the end
+of the program you may want to output the data somehow, but
+synchronizing this output is up to you and is beyond the scope of this
+manual.  (One good way to output a large multi-dimensional distributed
+array in MPI to a portable binary file is to use the free HDF5
+library; see the @uref{http://www.hdfgroup.org/, HDF home page}.)
+@cindex HDF5
+@cindex MPI I/O
+
+@c ------------------------------------------------------------
+@node MPI Data Distribution, Multi-dimensional MPI DFTs of Real Data, 2d MPI example, Distributed-memory FFTW with MPI
+@section MPI Data Distribution
+@cindex data distribution
+
+The most important concept to understand in using FFTW's MPI interface
+is the data distribution.  With a serial or multithreaded FFT, all of
+the inputs and outputs are stored as a single contiguous chunk of
+memory.  With a distributed-memory FFT, the inputs and outputs are
+broken into disjoint blocks, one per process.
+
+In particular, FFTW uses a @emph{1d block distribution} of the data,
+distributed along the @emph{first dimension}.  For example, if you
+want to perform a @twodims{100,200} complex DFT, distributed over 4
+processes, each process will get a @twodims{25,200} slice of the data.
+That is, process 0 will get rows 0 through 24, process 1 will get rows
+25 through 49, process 2 will get rows 50 through 74, and process 3
+will get rows 75 through 99.  If you take the same array but
+distribute it over 3 processes, then it is not evenly divisible so the
+different processes will have unequal chunks.  FFTW's default choice
+in this case is to assign 34 rows to processes 0 and 1, and 32 rows to
+process 2.
+@cindex block distribution
+
+
+FFTW provides several @samp{fftw_mpi_local_size} routines that you can
+call to find out what portion of an array is stored on the current
+process.  In most cases, you should use the default block sizes picked
+by FFTW, but it is also possible to specify your own block size.  For
+example, with a @twodims{100,200} array on three processes, you can
+tell FFTW to use a block size of 40, which would assign 40 rows to
+processes 0 and 1, and 20 rows to process 2.  FFTW's default is to
+divide the data equally among the processes if possible, and as best
+it can otherwise.  The rows are always assigned in ``rank order,''
+i.e. process 0 gets the first block of rows, then process 1, and so
+on.  (You can change this by using @code{MPI_Comm_split} to create a
+new communicator with re-ordered processes.)  However, you should
+always call the @samp{fftw_mpi_local_size} routines, if possible,
+rather than trying to predict FFTW's distribution choices.
+
+In particular, it is critical that you allocate the storage size that
+is returned by @samp{fftw_mpi_local_size}, which is @emph{not}
+necessarily the size of the local slice of the array.  The reason is
+that intermediate steps of FFTW's algorithms involve transposing the
+array and redistributing the data, so at these intermediate steps FFTW
+may require more local storage space (albeit always proportional to
+the total size divided by the number of processes).  The
+@samp{fftw_mpi_local_size} functions know how much storage is required
+for these intermediate steps and tell you the correct amount to
+allocate.
+
+@menu
+* Basic and advanced distribution interfaces::
+* Load balancing::
+* Transposed distributions::
+* One-dimensional distributions::
+@end menu
+
+@node Basic and advanced distribution interfaces, Load balancing, MPI Data Distribution, MPI Data Distribution
+@subsection Basic and advanced distribution interfaces
+
+As with the planner interface, the @samp{fftw_mpi_local_size}
+distribution interface is broken into basic and advanced
+(@samp{_many}) interfaces, where the latter allows you to specify the
+block size manually and also to request block sizes when computing
+multiple transforms simultaneously.  These functions are documented
+more exhaustively by the FFTW MPI Reference, but we summarize the
+basic ideas here using a couple of two-dimensional examples.
+
+For the @twodims{100,200} complex-DFT example, above, we would find
+the distribution by calling the following function in the basic
+interface:
+
+@example
+ptrdiff_t fftw_mpi_local_size_2d(ptrdiff_t n0, ptrdiff_t n1, MPI_Comm comm,
+                                 ptrdiff_t *local_n0, ptrdiff_t *local_0_start);
+@end example
+@findex fftw_mpi_local_size_2d
+
+Given the total size of the data to be transformed (here, @code{n0 =
+100} and @code{n1 = 200}) and an MPI communicator (@code{comm}), this
+function provides three numbers.
+
+First, it describes the shape of the local data: the current process
+should store a @code{local_n0} by @code{n1} slice of the overall
+dataset, in row-major order (@code{n1} dimension contiguous), starting
+at index @code{local_0_start}.  That is, if the total dataset is
+viewed as a @code{n0} by @code{n1} matrix, the current process should
+store the rows @code{local_0_start} to
+@code{local_0_start+local_n0-1}.  Obviously, if you are running with
+only a single MPI process, that process will store the entire array:
+@code{local_0_start} will be zero and @code{local_n0} will be
+@code{n0}.  @xref{Row-major Format}.
+@cindex row-major
+
+
+Second, the return value is the total number of data elements (e.g.,
+complex numbers for a complex DFT) that should be allocated for the
+input and output arrays on the current process (ideally with
+@code{fftw_malloc} or an @samp{fftw_alloc} function, to ensure optimal
+alignment).  It might seem that this should always be equal to
+@code{local_n0 * n1}, but this is @emph{not} the case.  FFTW's
+distributed FFT algorithms require data redistributions at
+intermediate stages of the transform, and in some circumstances this
+may require slightly larger local storage.  This is discussed in more
+detail below, under @ref{Load balancing}.
+@findex fftw_malloc
+@findex fftw_alloc_complex
+
+
+@cindex advanced interface
+The advanced-interface @samp{local_size} function for multidimensional
+transforms returns the same three things (@code{local_n0},
+@code{local_0_start}, and the total number of elements to allocate),
+but takes more inputs:
+
+@example
+ptrdiff_t fftw_mpi_local_size_many(int rnk, const ptrdiff_t *n,
+                                   ptrdiff_t howmany,
+                                   ptrdiff_t block0,
+                                   MPI_Comm comm,
+                                   ptrdiff_t *local_n0,
+                                   ptrdiff_t *local_0_start);
+@end example
+@findex fftw_mpi_local_size_many
+
+The two-dimensional case above corresponds to @code{rnk = 2} and an
+array @code{n} of length 2 with @code{n[0] = n0} and @code{n[1] = n1}.
+This routine is for any @code{rnk > 1}; one-dimensional transforms
+have their own interface because they work slightly differently, as
+discussed below.
+
+First, the advanced interface allows you to perform multiple
+transforms at once, of interleaved data, as specified by the
+@code{howmany} parameter.  (@code{hoamany} is 1 for a single
+transform.)
+
+Second, here you can specify your desired block size in the @code{n0}
+dimension, @code{block0}.  To use FFTW's default block size, pass
+@code{FFTW_MPI_DEFAULT_BLOCK} (0) for @code{block0}.  Otherwise, on
+@code{P} processes, FFTW will return @code{local_n0} equal to
+@code{block0} on the first @code{P / block0} processes (rounded down),
+return @code{local_n0} equal to @code{n0 - block0 * (P / block0)} on
+the next process, and @code{local_n0} equal to zero on any remaining
+processes.  In general, we recommend using the default block size
+(which corresponds to @code{n0 / P}, rounded up).
+@ctindex FFTW_MPI_DEFAULT_BLOCK
+@cindex block distribution
+
+
+For example, suppose you have @code{P = 4} processes and @code{n0 =
+21}.  The default will be a block size of @code{6}, which will give
+@code{local_n0 = 6} on the first three processes and @code{local_n0 =
+3} on the last process.  Instead, however, you could specify
+@code{block0 = 5} if you wanted, which would give @code{local_n0 = 5}
+on processes 0 to 2, @code{local_n0 = 6} on process 3.  (This choice,
+while it may look superficially more ``balanced,'' has the same
+critical path as FFTW's default but requires more communications.)
+
+@node Load balancing, Transposed distributions, Basic and advanced distribution interfaces, MPI Data Distribution
+@subsection Load balancing
+@cindex load balancing
+
+Ideally, when you parallelize a transform over some @math{P}
+processes, each process should end up with work that takes equal time.
+Otherwise, all of the processes end up waiting on whichever process is
+slowest.  This goal is known as ``load balancing.''  In this section,
+we describe the circumstances under which FFTW is able to load-balance
+well, and in particular how you should choose your transform size in
+order to load balance.
+
+Load balancing is especially difficult when you are parallelizing over
+heterogeneous machines; for example, if one of your processors is a
+old 486 and another is a Pentium IV, obviously you should give the
+Pentium more work to do than the 486 since the latter is much slower.
+FFTW does not deal with this problem, however---it assumes that your
+processes run on hardware of comparable speed, and that the goal is
+therefore to divide the problem as equally as possible.
+
+For a multi-dimensional complex DFT, FFTW can divide the problem
+equally among the processes if: (i) the @emph{first} dimension
+@code{n0} is divisible by @math{P}; and (ii), the @emph{product} of
+the subsequent dimensions is divisible by @math{P}.  (For the advanced
+interface, where you can specify multiple simultaneous transforms via
+some ``vector'' length @code{howmany}, a factor of @code{howmany} is
+included in the product of the subsequent dimensions.)
+
+For a one-dimensional complex DFT, the length @code{N} of the data
+should be divisible by @math{P} @emph{squared} to be able to divide
+the problem equally among the processes.
+
+@node Transposed distributions, One-dimensional distributions, Load balancing, MPI Data Distribution
+@subsection Transposed distributions
+
+Internally, FFTW's MPI transform algorithms work by first computing
+transforms of the data local to each process, then by globally
+@emph{transposing} the data in some fashion to redistribute the data
+among the processes, transforming the new data local to each process,
+and transposing back.  For example, a two-dimensional @code{n0} by
+@code{n1} array, distributed across the @code{n0} dimension, is
+transformd by: (i) transforming the @code{n1} dimension, which are
+local to each process; (ii) transposing to an @code{n1} by @code{n0}
+array, distributed across the @code{n1} dimension; (iii) transforming
+the @code{n0} dimension, which is now local to each process; (iv)
+transposing back.
+@cindex transpose
+
+
+However, in many applications it is acceptable to compute a
+multidimensional DFT whose results are produced in transposed order
+(e.g., @code{n1} by @code{n0} in two dimensions).  This provides a
+significant performance advantage, because it means that the final
+transposition step can be omitted.  FFTW supports this optimization,
+which you specify by passing the flag @code{FFTW_MPI_TRANSPOSED_OUT}
+to the planner routines.  To compute the inverse transform of
+transposed output, you specify @code{FFTW_MPI_TRANSPOSED_IN} to tell
+it that the input is transposed.  In this section, we explain how to
+interpret the output format of such a transform.
+@ctindex FFTW_MPI_TRANSPOSED_OUT
+@ctindex FFTW_MPI_TRANSPOSED_IN
+
+
+Suppose you have are transforming multi-dimensional data with (at
+least two) dimensions @ndims{}.  As always, it is distributed along
+the first dimension @dimk{0}.  Now, if we compute its DFT with the
+@code{FFTW_MPI_TRANSPOSED_OUT} flag, the resulting output data are stored
+with the first @emph{two} dimensions transposed: @ndimstrans{},
+distributed along the @dimk{1} dimension.  Conversely, if we take the
+@ndimstrans{} data and transform it with the
+@code{FFTW_MPI_TRANSPOSED_IN} flag, then the format goes back to the
+original @ndims{} array.
+
+There are two ways to find the portion of the transposed array that
+resides on the current process.  First, you can simply call the
+appropriate @samp{local_size} function, passing @ndimstrans{} (the
+transposed dimensions).  This would mean calling the @samp{local_size}
+function twice, once for the transposed and once for the
+non-transposed dimensions.  Alternatively, you can call one of the
+@samp{local_size_transposed} functions, which returns both the
+non-transposed and transposed data distribution from a single call.
+For example, for a 3d transform with transposed output (or input), you
+might call:
+
+@example
+ptrdiff_t fftw_mpi_local_size_3d_transposed(
+                ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2, MPI_Comm comm,
+                ptrdiff_t *local_n0, ptrdiff_t *local_0_start,
+                ptrdiff_t *local_n1, ptrdiff_t *local_1_start);
+@end example
+@findex fftw_mpi_local_size_3d_transposed
+
+Here, @code{local_n0} and @code{local_0_start} give the size and
+starting index of the @code{n0} dimension for the
+@emph{non}-transposed data, as in the previous sections.  For
+@emph{transposed} data (e.g. the output for
+@code{FFTW_MPI_TRANSPOSED_OUT}), @code{local_n1} and
+@code{local_1_start} give the size and starting index of the @code{n1}
+dimension, which is the first dimension of the transposed data
+(@code{n1} by @code{n0} by @code{n2}).
+
+(Note that @code{FFTW_MPI_TRANSPOSED_IN} is completely equivalent to
+performing @code{FFTW_MPI_TRANSPOSED_OUT} and passing the first two
+dimensions to the planner in reverse order, or vice versa.  If you
+pass @emph{both} the @code{FFTW_MPI_TRANSPOSED_IN} and
+@code{FFTW_MPI_TRANSPOSED_OUT} flags, it is equivalent to swapping the
+first two dimensions passed to the planner and passing @emph{neither}
+flag.)
+
+@node One-dimensional distributions,  , Transposed distributions, MPI Data Distribution
+@subsection One-dimensional distributions
+
+For one-dimensional distributed DFTs using FFTW, matters are slightly
+more complicated because the data distribution is more closely tied to
+how the algorithm works.  In particular, you can no longer pass an
+arbitrary block size and must accept FFTW's default; also, the block
+sizes may be different for input and output.  Also, the data
+distribution depends on the flags and transform direction, in order
+for forward and backward transforms to work correctly.
+
+@example
+ptrdiff_t fftw_mpi_local_size_1d(ptrdiff_t n0, MPI_Comm comm,
+                int sign, unsigned flags,
+                ptrdiff_t *local_ni, ptrdiff_t *local_i_start,
+                ptrdiff_t *local_no, ptrdiff_t *local_o_start);
+@end example
+@findex fftw_mpi_local_size_1d
+
+This function computes the data distribution for a 1d transform of
+size @code{n0} with the given transform @code{sign} and @code{flags}.
+Both input and output data use block distributions.  The input on the
+current process will consist of @code{local_ni} numbers starting at
+index @code{local_i_start}; e.g. if only a single process is used,
+then @code{local_ni} will be @code{n0} and @code{local_i_start} will
+be @code{0}.  Similarly for the output, with @code{local_no} numbers
+starting at index @code{local_o_start}.  The return value of
+@code{fftw_mpi_local_size_1d} will be the total number of elements to
+allocate on the current process (which might be slightly larger than
+the local size due to intermediate steps in the algorithm).
+
+As mentioned above (@pxref{Load balancing}), the data will be divided
+equally among the processes if @code{n0} is divisible by the
+@emph{square} of the number of processes.  In this case,
+@code{local_ni} will equal @code{local_no}.  Otherwise, they may be
+different.
+
+For some applications, such as convolutions, the order of the output
+data is irrelevant.  In this case, performance can be improved by
+specifying that the output data be stored in an FFTW-defined
+``scrambled'' format.  (In particular, this is the analogue of
+transposed output in the multidimensional case: scrambled output saves
+a communications step.)  If you pass @code{FFTW_MPI_SCRAMBLED_OUT} in
+the flags, then the output is stored in this (undocumented) scrambled
+order.  Conversely, to perform the inverse transform of data in
+scrambled order, pass the @code{FFTW_MPI_SCRAMBLED_IN} flag.
+@ctindex FFTW_MPI_SCRAMBLED_OUT
+@ctindex FFTW_MPI_SCRAMBLED_IN
+
+
+In MPI FFTW, only composite sizes @code{n0} can be parallelized; we
+have not yet implemented a parallel algorithm for large prime sizes.
+
+@c ------------------------------------------------------------
+@node Multi-dimensional MPI DFTs of Real Data, Other Multi-dimensional Real-data MPI Transforms, MPI Data Distribution, Distributed-memory FFTW with MPI
+@section Multi-dimensional MPI DFTs of Real Data
+
+FFTW's MPI interface also supports multi-dimensional DFTs of real
+data, similar to the serial r2c and c2r interfaces.  (Parallel
+one-dimensional real-data DFTs are not currently supported; you must
+use a complex transform and set the imaginary parts of the inputs to
+zero.)
+
+The key points to understand for r2c and c2r MPI transforms (compared
+to the MPI complex DFTs or the serial r2c/c2r transforms), are:
+
+@itemize @bullet
+
+@item
+Just as for serial transforms, r2c/c2r DFTs transform @ndims{} real
+data to/from @ndimshalf{} complex data: the last dimension of the
+complex data is cut in half (rounded down), plus one.  As for the
+serial transforms, the sizes you pass to the @samp{plan_dft_r2c} and
+@samp{plan_dft_c2r} are the @ndims{} dimensions of the real data.
+
+@item
+@cindex padding
+Although the real data is @emph{conceptually} @ndims{}, it is
+@emph{physically} stored as an @ndimspad{} array, where the last
+dimension has been @emph{padded} to make it the same size as the
+complex output.  This is much like the in-place serial r2c/c2r
+interface (@pxref{Multi-Dimensional DFTs of Real Data}), except that
+in MPI the padding is required even for out-of-place data.  The extra
+padding numbers are ignored by FFTW (they are @emph{not} like
+zero-padding the transform to a larger size); they are only used to
+determine the data layout.
+
+@item
+@cindex data distribution
+The data distribution in MPI for @emph{both} the real and complex data
+is determined by the shape of the @emph{complex} data.  That is, you
+call the appropriate @samp{local size} function for the @ndimshalf{}
+complex data, and then use the @emph{same} distribution for the real
+data except that the last complex dimension is replaced by a (padded)
+real dimension of twice the length.
+
+@end itemize
+
+For example suppose we are performing an out-of-place r2c transform of
+@threedims{L,M,N} real data [padded to @threedims{L,M,2(N/2+1)}],
+resulting in @threedims{L,M,N/2+1} complex data.  Similar to the
+example in @ref{2d MPI example}, we might do something like:
+
+@example
+#include <fftw3-mpi.h>
+
+int main(int argc, char **argv)
+@{
+    const ptrdiff_t L = ..., M = ..., N = ...;
+    fftw_plan plan;
+    double *rin;
+    fftw_complex *cout;
+    ptrdiff_t alloc_local, local_n0, local_0_start, i, j, k;
+
+    MPI_Init(&argc, &argv);
+    fftw_mpi_init();
+
+    /* @r{get local data size and allocate} */
+    alloc_local = fftw_mpi_local_size_3d(L, M, N/2+1, MPI_COMM_WORLD,
+                                         &local_n0, &local_0_start);
+    rin = fftw_alloc_real(2 * alloc_local);
+    cout = fftw_alloc_complex(alloc_local);
+
+    /* @r{create plan for out-of-place r2c DFT} */
+    plan = fftw_mpi_plan_dft_r2c_3d(L, M, N, rin, cout, MPI_COMM_WORLD,
+                                    FFTW_MEASURE);
+
+    /* @r{initialize rin to some function} my_func(x,y,z) */
+    for (i = 0; i < local_n0; ++i)
+       for (j = 0; j < M; ++j)
+         for (k = 0; k < N; ++k)
+       rin[(i*M + j) * (2*(N/2+1)) + k] = my_func(local_0_start+i, j, k);
+
+    /* @r{compute transforms as many times as desired} */
+    fftw_execute(plan);
+
+    fftw_destroy_plan(plan);
+
+    MPI_Finalize();
+@}
+@end example
+
+@findex fftw_alloc_real
+@cindex row-major
+Note that we allocated @code{rin} using @code{fftw_alloc_real} with an
+argument of @code{2 * alloc_local}: since @code{alloc_local} is the
+number of @emph{complex} values to allocate, the number of @emph{real}
+values is twice as many.  The @code{rin} array is then
+@threedims{local_n0,M,2(N/2+1)} in row-major order, so its
+@code{(i,j,k)} element is at the index @code{(i*M + j) * (2*(N/2+1)) +
+k} (@pxref{Multi-dimensional Array Format }).
+
+@cindex transpose
+@ctindex FFTW_TRANSPOSED_OUT
+@ctindex FFTW_TRANSPOSED_IN
+As for the complex transforms, improved performance can be obtained by
+specifying that the output is the transpose of the input or vice versa
+(@pxref{Transposed distributions}).  In our @threedims{L,M,N} r2c
+example, including @code{FFTW_TRANSPOSED_OUT} in the flags means that
+the input would be a padded @threedims{L,M,2(N/2+1)} real array
+distributed over the @code{L} dimension, while the output would be a
+@threedims{M,L,N/2+1} complex array distributed over the @code{M}
+dimension.  To perform the inverse c2r transform with the same data
+distributions, you would use the @code{FFTW_TRANSPOSED_IN} flag.
+
+@c ------------------------------------------------------------
+@node Other Multi-dimensional Real-data MPI Transforms, FFTW MPI Transposes, Multi-dimensional MPI DFTs of Real Data, Distributed-memory FFTW with MPI
+@section Other multi-dimensional Real-Data MPI Transforms
+
+@cindex r2r
+FFTW's MPI interface also supports multi-dimensional @samp{r2r}
+transforms of all kinds supported by the serial interface
+(e.g. discrete cosine and sine transforms, discrete Hartley
+transforms, etc.).  Only multi-dimensional @samp{r2r} transforms, not
+one-dimensional transforms, are currently parallelized.
+
+@tindex fftw_r2r_kind
+These are used much like the multidimensional complex DFTs discussed
+above, except that the data is real rather than complex, and one needs
+to pass an r2r transform kind (@code{fftw_r2r_kind}) for each
+dimension as in the serial FFTW (@pxref{More DFTs of Real Data}).
+
+For example, one might perform a two-dimensional @twodims{L,M} that is
+an REDFT10 (DCT-II) in the first dimension and an RODFT10 (DST-II) in
+the second dimension with code like:
+
+@example
+    const ptrdiff_t L = ..., M = ...;
+    fftw_plan plan;
+    double *data;
+    ptrdiff_t alloc_local, local_n0, local_0_start, i, j;
+
+    /* @r{get local data size and allocate} */
+    alloc_local = fftw_mpi_local_size_2d(L, M, MPI_COMM_WORLD,
+                                         &local_n0, &local_0_start);
+    data = fftw_alloc_real(alloc_local);
+
+    /* @r{create plan for in-place REDFT10 x RODFT10} */
+    plan = fftw_mpi_plan_r2r_2d(L, M, data, data, MPI_COMM_WORLD,
+                                FFTW_REDFT10, FFTW_RODFT10, FFTW_MEASURE);
+
+    /* @r{initialize data to some function} my_function(x,y) */
+    for (i = 0; i < local_n0; ++i) for (j = 0; j < M; ++j)
+       data[i*M + j] = my_function(local_0_start + i, j);
+
+    /* @r{compute transforms, in-place, as many times as desired} */
+    fftw_execute(plan);
+
+    fftw_destroy_plan(plan);
+@end example
+
+@findex fftw_alloc_real
+Notice that we use the same @samp{local_size} functions as we did for
+complex data, only now we interpret the sizes in terms of real rather
+than complex values, and correspondingly use @code{fftw_alloc_real}.
+
+@c ------------------------------------------------------------
+@node FFTW MPI Transposes, FFTW MPI Wisdom, Other Multi-dimensional Real-data MPI Transforms, Distributed-memory FFTW with MPI
+@section FFTW MPI Transposes
+@cindex transpose
+
+The FFTW's MPI Fourier transforms rely on one or more @emph{global
+transposition} step for their communications.  For example, the
+multidimensional transforms work by transforming along some
+dimensions, then transposing to make the first dimension local and
+transforming that, then transposing back.  Because global
+transposition of a block-distributed matrix has many other potential
+uses besides FFTs, FFTW's transpose routines can be called directly,
+as documented in this section. 
+
+@menu
+* Basic distributed-transpose interface::
+* Advanced distributed-transpose interface::
+* An improved replacement for MPI_Alltoall::
+@end menu
+
+@node Basic distributed-transpose interface, Advanced distributed-transpose interface, FFTW MPI Transposes, FFTW MPI Transposes
+@subsection Basic distributed-transpose interface
+
+In particular, suppose that we have an @code{n0} by @code{n1} array in
+row-major order, block-distributed across the @code{n0} dimension.  To
+transpose this into an @code{n1} by @code{n0} array block-distributed
+across the @code{n1} dimension, we would create a plan by calling the
+following function:
+
+@example
+fftw_plan fftw_mpi_plan_transpose(ptrdiff_t n0, ptrdiff_t n1,
+                                  double *in, double *out,
+                                  MPI_Comm comm, unsigned flags);
+@end example
+@findex fftw_mpi_plan_transpose
+
+The input and output arrays (@code{in} and @code{out}) can be the
+same.  The transpose is actually executed by calling
+@code{fftw_execute} on the plan, as usual.
+@findex fftw_execute
+
+
+The @code{flags} are the usual FFTW planner flags, but support
+two additional flags: @code{FFTW_MPI_TRANSPOSED_OUT} and/or
+@code{FFTW_MPI_TRANSPOSED_IN}.  What these flags indicate, for
+transpose plans, is that the output and/or input, respectively, are
+@emph{locally} transposed.  That is, on each process input data is
+normally stored as a @code{local_n0} by @code{n1} array in row-major
+order, but for an @code{FFTW_MPI_TRANSPOSED_IN} plan the input data is
+stored as @code{n1} by @code{local_n0} in row-major order.  Similarly,
+@code{FFTW_MPI_TRANSPOSED_OUT} means that the output is @code{n0} by
+@code{local_n1} instead of @code{local_n1} by @code{n0}.
+@ctindex FFTW_MPI_TRANSPOSED_OUT
+@ctindex FFTW_MPI_TRANSPOSED_IN
+
+
+To determine the local size of the array on each process before and
+after the transpose, as well as the amount of storage that must be
+allocated, one should call @code{fftw_mpi_local_size_2d_transposed},
+just as for a 2d DFT as described in the previous section:
+@cindex data distribution
+
+@example
+ptrdiff_t fftw_mpi_local_size_2d_transposed
+                (ptrdiff_t n0, ptrdiff_t n1, MPI_Comm comm,
+                 ptrdiff_t *local_n0, ptrdiff_t *local_0_start,
+                 ptrdiff_t *local_n1, ptrdiff_t *local_1_start);
+@end example
+@findex fftw_mpi_local_size_2d_transposed
+
+Again, the return value is the local storage to allocate, which in
+this case is the number of @emph{real} (@code{double}) values rather
+than complex numbers as in the previous examples.
+
+@node Advanced distributed-transpose interface, An improved replacement for MPI_Alltoall, Basic distributed-transpose interface, FFTW MPI Transposes
+@subsection Advanced distributed-transpose interface
+
+The above routines are for a transpose of a matrix of numbers (of type
+@code{double}), using FFTW's default block sizes.  More generally, one
+can perform transposes of @emph{tuples} of numbers, with
+user-specified block sizes for the input and output:
+
+@example
+fftw_plan fftw_mpi_plan_many_transpose
+                (ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t howmany,
+                 ptrdiff_t block0, ptrdiff_t block1,
+                 double *in, double *out, MPI_Comm comm, unsigned flags);
+@end example
+@findex fftw_mpi_plan_many_transpose
+
+In this case, one is transposing an @code{n0} by @code{n1} matrix of
+@code{howmany}-tuples (e.g. @code{howmany = 2} for complex numbers).
+The input is distributed along the @code{n0} dimension with block size
+@code{block0}, and the @code{n1} by @code{n0} output is distributed
+along the @code{n1} dimension with block size @code{block1}.  If
+@code{FFTW_MPI_DEFAULT_BLOCK} (0) is passed for a block size then FFTW
+uses its default block size.  To get the local size of the data on
+each process, you should then call @code{fftw_mpi_local_size_many_transposed}.
+@ctindex FFTW_MPI_DEFAULT_BLOCK
+@findex fftw_mpi_local_size_many_transposed
+
+@node An improved replacement for MPI_Alltoall,  , Advanced distributed-transpose interface, FFTW MPI Transposes
+@subsection An improved replacement for MPI_Alltoall
+
+We close this section by noting that FFTW's MPI transpose routines can
+be thought of as a generalization for the @code{MPI_Alltoall} function
+(albeit only for floating-point types), and in some circumstances can
+function as an improved replacement.
+@findex MPI_Alltoall
+
+
+@code{MPI_Alltoall} is defined by the MPI standard as:
+
+@example
+int MPI_Alltoall(void *sendbuf, int sendcount, MPI_Datatype sendtype, 
+                 void *recvbuf, int recvcnt, MPI_Datatype recvtype, 
+                 MPI_Comm comm);
+@end example
+
+In particular, for @code{double*} arrays @code{in} and @code{out},
+consider the call:
+
+@example
+MPI_Alltoall(in, howmany, MPI_DOUBLE, out, howmany MPI_DOUBLE, comm);
+@end example
+
+This is completely equivalent to:
+
+@example
+MPI_Comm_size(comm, &P);
+plan = fftw_mpi_plan_many_transpose(P, P, howmany, 1, 1, in, out, comm, FFTW_ESTIMATE);
+fftw_execute(plan);
+fftw_destroy_plan(plan);
+@end example
+
+That is, computing a @twodims{P,P} transpose on @code{P} processes,
+with a block size of 1, is just a standard all-to-all communication.
+
+However, using the FFTW routine instead of @code{MPI_Alltoall} may
+have certain advantages.  First of all, FFTW's routine can operate
+in-place (@code{in == out}) whereas @code{MPI_Alltoall} can only
+operate out-of-place.
+@cindex in-place
+
+
+Second, even for out-of-place plans, FFTW's routine may be faster,
+especially if you need to perform the all-to-all communication many
+times and can afford to use @code{FFTW_MEASURE} or
+@code{FFTW_PATIENT}.  It should certainly be no slower, not including
+the time to create the plan, since one of the possible algorithms that
+FFTW uses for an out-of-place transpose @emph{is} simply to call
+@code{MPI_Alltoall}.  However, FFTW also considers several other
+possible algorithms that, depending on your MPI implementation and
+your hardware, may be faster.
+@ctindex FFTW_MEASURE
+@ctindex FFTW_PATIENT
+
+@c ------------------------------------------------------------
+@node FFTW MPI Wisdom, Avoiding MPI Deadlocks, FFTW MPI Transposes, Distributed-memory FFTW with MPI
+@section FFTW MPI Wisdom
+@cindex wisdom
+@cindex saving plans to disk
+
+FFTW's ``wisdom'' facility (@pxref{Words of Wisdom-Saving Plans}) can
+be used to save MPI plans as well as to save uniprocessor plans.
+However, for MPI there are several unavoidable complications.
+
+@cindex MPI I/O
+First, the MPI standard does not guarantee that every process can
+perform file I/O (at least, not using C stdio routines)---in general,
+we may only assume that process 0 is capable of I/O.@footnote{In fact,
+even this assumption is not technically guaranteed by the standard,
+although it seems to be universal in actual MPI implementations and is
+widely assumed by MPI-using software.  Technically, you need to query
+the @code{MPI_IO} attribute of @code{MPI_COMM_WORLD} with
+@code{MPI_Attr_get}.  If this attribute is @code{MPI_PROC_NULL}, no
+I/O is possible.  If it is @code{MPI_ANY_SOURCE}, any process can
+perform I/O.  Otherwise, it is the rank of a process that can perform
+I/O ... but since it is not guaranteed to yield the @emph{same} rank
+on all processes, you have to do an @code{MPI_Allreduce} of some kind
+if you want all processes to agree about which is going to do I/O.
+And even then, the standard only guarantees that this process can
+perform output, but not input. See e.g. @cite{Parallel Programming
+with MPI} by P. S. Pacheco, section 8.1.3.  Needless to say, in our
+experience virtually no MPI programmers worry about this.} So, if we
+want to export the wisdom from a single process to a file, we must
+first export the wisdom to a string, then send it to process 0, then
+write it to a file.
+
+Second, in principle we may want to have separate wisdom for every
+process, since in general the processes may run on different hardware
+even for a single MPI program.  However, in practice FFTW's MPI code
+is designed for the case of homogeneous hardware (@pxref{Load
+balancing}), and in this case it is convenient to use the same wisdom
+for every process.  Thus, we need a mechanism to synchronize the wisdom.
+
+To address both of these problems, FFTW provides the following two
+functions:
+
+@example
+void fftw_mpi_broadcast_wisdom(MPI_Comm comm);
+void fftw_mpi_gather_wisdom(MPI_Comm comm);
+@end example
+@findex fftw_mpi_gather_wisdom
+@findex fftw_mpi_broadcast_wisdom
+
+Given a communicator @code{comm}, @code{fftw_mpi_broadcast_wisdom}
+will broadcast the wisdom from process 0 to all other processes.
+Conversely, @code{fftw_mpi_gather_wisdom} will collect wisdom from all
+processes onto process 0.  (If the plans created for the same problem
+by different processes are not the same, @code{fftw_mpi_gather_wisdom}
+will arbitrarily choose one of the plans.)  Both of these functions
+may result in suboptimal plans for different processes if the
+processes are running on non-identical hardware.  Both of these
+functions are @emph{collective} calls, which means that they must be
+executed by all processes in the communicator.
+@cindex collective function
+
+
+So, for example, a typical code snippet to import wisdom from a file
+and use it on all processes would be:
+
+@example
+@{
+    int rank;
+
+    fftw_mpi_init();
+    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+    if (rank == 0) fftw_import_wisdom_from_filename("mywisdom");
+    fftw_mpi_broadcast_wisdom(MPI_COMM_WORLD);
+@}
+@end example
+
+(Note that we must call @code{fftw_mpi_init} before importing any
+wisdom that might contain MPI plans.)  Similarly, a typical code
+snippet to export wisdom from all processes to a file is:
+@findex fftw_mpi_init
+
+@example
+@{
+    int rank;
+
+    fftw_mpi_gather_wisdom(MPI_COMM_WORLD);
+    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+    if (rank == 0) fftw_export_wisdom_to_filename("mywisdom");
+@}
+@end example
+
+@c ------------------------------------------------------------
+@node Avoiding MPI Deadlocks, FFTW MPI Performance Tips, FFTW MPI Wisdom, Distributed-memory FFTW with MPI
+@section Avoiding MPI Deadlocks
+@cindex deadlock
+
+An MPI program can @emph{deadlock} if one process is waiting for a
+message from another process that never gets sent.  To avoid deadlocks
+when using FFTW's MPI routines, it is important to know which
+functions are @emph{collective}: that is, which functions must
+@emph{always} be called in the @emph{same order} from @emph{every}
+process in a given communicator.  (For example, @code{MPI_Barrier} is
+the canonical example of a collective function in the MPI standard.)
+@cindex collective function
+@findex MPI_Barrier
+
+
+The functions in FFTW that are @emph{always} collective are: every
+function beginning with @samp{fftw_mpi_plan}, as well as
+@code{fftw_mpi_broadcast_wisdom} and @code{fftw_mpi_gather_wisdom}.
+Also, the following functions from the ordinary FFTW interface are
+collective when they are applied to a plan created by an
+@samp{fftw_mpi_plan} function: @code{fftw_execute},
+@code{fftw_destroy_plan}, and @code{fftw_flops}.
+@findex fftw_execute
+@findex fftw_destroy_plan
+@findex fftw_flops
+
+@c ------------------------------------------------------------
+@node FFTW MPI Performance Tips, Combining MPI and Threads, Avoiding MPI Deadlocks, Distributed-memory FFTW with MPI
+@section FFTW MPI Performance Tips
+
+In this section, we collect a few tips on getting the best performance
+out of FFTW's MPI transforms.
+
+First, because of the 1d block distribution, FFTW's parallelization is
+currently limited by the size of the first dimension.
+(Multidimensional block distributions may be supported by a future
+version.) More generally, you should ideally arrange the dimensions so
+that FFTW can divide them equally among the processes. @xref{Load
+balancing}.
+@cindex block distribution
+@cindex load balancing
+
+
+Second, if it is not too inconvenient, you should consider working
+with transposed output for multidimensional plans, as this saves a
+considerable amount of communications.  @xref{Transposed distributions}.
+@cindex transpose
+
+
+Third, the fastest choices are generally either an in-place transform
+or an out-of-place transform with the @code{FFTW_DESTROY_INPUT} flag
+(which allows the input array to be used as scratch space).  In-place
+is especially beneficial if the amount of data per process is large.
+@ctindex FFTW_DESTROY_INPUT
+
+
+Fourth, if you have multiple arrays to transform at once, rather than
+calling FFTW's MPI transforms several times it usually seems to be
+faster to interleave the data and use the advanced interface.  (This
+groups the communications together instead of requiring separate
+messages for each transform.)
+
+@c ------------------------------------------------------------
+@node Combining MPI and Threads, FFTW MPI Reference, FFTW MPI Performance Tips, Distributed-memory FFTW with MPI
+@section Combining MPI and Threads
+@cindex threads
+
+In certain cases, it may be advantageous to combine MPI
+(distributed-memory) and threads (shared-memory) parallelization.
+FFTW supports this, with certain caveats.  For example, if you have a
+cluster of 4-processor shared-memory nodes, you may want to use
+threads within the nodes and MPI between the nodes, instead of MPI for
+all parallelization.
+
+In particular, it is possible to seamlessly combine the MPI FFTW
+routines with the multi-threaded FFTW routines (@pxref{Multi-threaded
+FFTW}). However, some care must be taken in the initialization code,
+which should look something like this:
+
+@example
+int threads_ok;
+
+int main(int argc, char **argv)
+@{
+    int provided;
+    MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
+    threads_ok = provided >= MPI_THREAD_FUNNELED;
+
+    if (threads_ok) threads_ok = fftw_init_threads();
+    fftw_mpi_init();
+
+    ...
+    if (threads_ok) fftw_plan_with_nthreads(...);
+    ...
+    
+    MPI_Finalize();
+@}
+@end example
+@findex fftw_mpi_init
+@findex fftw_init_threads
+@findex fftw_plan_with_nthreads
+
+First, note that instead of calling @code{MPI_Init}, you should call
+@code{MPI_Init_threads}, which is the initialization routine defined
+by the MPI-2 standard to indicate to MPI that your program will be
+multithreaded.  We pass @code{MPI_THREAD_FUNNELED}, which indicates
+that we will only call MPI routines from the main thread.  (FFTW will
+launch additional threads internally, but the extra threads will not
+call MPI code.)  (You may also pass @code{MPI_THREAD_SERIALIZED} or
+@code{MPI_THREAD_MULTIPLE}, which requests additional multithreading
+support from the MPI implementation, but this is not required by
+FFTW.)  The @code{provided} parameter returns what level of threads
+support is actually supported by your MPI implementation; this
+@emph{must} be at least @code{MPI_THREAD_FUNNELED} if you want to call
+the FFTW threads routines, so we define a global variable
+@code{threads_ok} to record this.  You should only call
+@code{fftw_init_threads} or @code{fftw_plan_with_nthreads} if
+@code{threads_ok} is true.  For more information on thread safety in
+MPI, see the
+@uref{http://www.mpi-forum.org/docs/mpi-20-html/node162.htm, MPI and
+Threads} section of the MPI-2 standard.
+@cindex thread safety
+
+
+Second, we must call @code{fftw_init_threads} @emph{before}
+@code{fftw_mpi_init}.  This is critical for technical reasons having
+to do with how FFTW initializes its list of algorithms.
+
+Then, if you call @code{fftw_plan_with_nthreads(N)}, @emph{every} MPI
+process will launch (up to) @code{N} threads to parallelize its transforms.
+
+For example, in the hypothetical cluster of 4-processor nodes, you
+might wish to launch only a single MPI process per node, and then call
+@code{fftw_plan_with_nthreads(4)} on each process to use all
+processors in the nodes.
+
+This may or may not be faster than simply using as many MPI processes
+as you have processors, however.  On the one hand, using threads
+within a node eliminates the need for explicit message passing within
+the node.  On the other hand, FFTW's transpose routines are not
+multi-threaded, and this means that the communications that do take
+place will not benefit from parallelization within the node.
+Moreover, many MPI implementations already have optimizations to
+exploit shared memory when it is available, so adding the
+multithreaded FFTW on top of this may be superfluous.
+@cindex transpose
+
+@c ------------------------------------------------------------
+@node FFTW MPI Reference, FFTW MPI Fortran Interface, Combining MPI and Threads, Distributed-memory FFTW with MPI
+@section FFTW MPI Reference
+
+This chapter provides a complete reference to all FFTW MPI functions,
+datatypes, and constants.  See also @ref{FFTW Reference} for information
+on functions and types in common with the serial interface.
+
+@menu
+* MPI Files and Data Types::
+* MPI Initialization::
+* Using MPI Plans::
+* MPI Data Distribution Functions::
+* MPI Plan Creation::
+* MPI Wisdom Communication::
+@end menu
+
+@node MPI Files and Data Types, MPI Initialization, FFTW MPI Reference, FFTW MPI Reference
+@subsection MPI Files and Data Types
+
+All programs using FFTW's MPI support should include its header file:
+
+@example
+#include <fftw3-mpi.h>
+@end example
+
+Note that this header file includes the serial-FFTW @code{fftw3.h}
+header file, and also the @code{mpi.h} header file for MPI, so you
+need not include those files separately.
+
+You must also link to @emph{both} the FFTW MPI library and to the
+serial FFTW library.  On Unix, this means adding @code{-lfftw3_mpi
+-lfftw3 -lm} at the end of the link command.
+
+@cindex precision
+Different precisions are handled as in the serial interface:
+@xref{Precision}.  That is, @samp{fftw_} functions become
+@code{fftwf_} (in single precision) etcetera, and the libraries become
+@code{-lfftw3f_mpi -lfftw3f -lm} etcetera on Unix.  Long-double
+precision is supported in MPI, but quad precision (@samp{fftwq_}) is
+not due to the lack of MPI support for this type.
+
+@node MPI Initialization, Using MPI Plans, MPI Files and Data Types, FFTW MPI Reference
+@subsection MPI Initialization
+
+Before calling any other FFTW MPI (@samp{fftw_mpi_}) function, and
+before importing any wisdom for MPI problems, you must call:
+
+@findex fftw_mpi_init
+@example
+void fftw_mpi_init(void);
+@end example
+
+@findex fftw_init_threads
+If FFTW threads support is used, however, @code{fftw_mpi_init} should
+be called @emph{after} @code{fftw_init_threads} (@pxref{Combining MPI
+and Threads}).  Calling @code{fftw_mpi_init} additional times (before
+@code{fftw_mpi_cleanup}) has no effect.
+
+
+If you want to deallocate all persistent data and reset FFTW to the
+pristine state it was in when you started your program, you can call:
+
+@findex fftw_mpi_cleanup
+@example
+void fftw_mpi_cleanup(void);
+@end example
+
+@findex fftw_cleanup
+(This calls @code{fftw_cleanup}, so you need not call the serial
+cleanup routine too, although it is safe to do so.)  After calling
+@code{fftw_mpi_cleanup}, all existing plans become undefined, and you
+should not attempt to execute or destroy them.  You must call
+@code{fftw_mpi_init} again after @code{fftw_mpi_cleanup} if you want
+to resume using the MPI FFTW routines.
+
+@node Using MPI Plans, MPI Data Distribution Functions, MPI Initialization, FFTW MPI Reference
+@subsection Using MPI Plans
+
+Once an MPI plan is created, you can execute and destroy it using
+@code{fftw_execute}, @code{fftw_destroy_plan}, and the other functions
+in the serial interface that operate on generic plans (@pxref{Using
+Plans}).  
+
+@cindex collective function
+@cindex MPI communicator
+The @code{fftw_execute} and @code{fftw_destroy_plan} functions, applied to
+MPI plans, are @emph{collective} calls: they must be called for all processes
+in the communicator that was used to create the plan.
+
+@cindex new-array execution
+You must @emph{not} use the serial new-array plan-execution functions
+@code{fftw_execute_dft} and so on (@pxref{New-array Execute
+Functions}) with MPI plans.  Such functions are specialized to the
+problem type, and there are specific new-array execute functions for MPI plans:
+
+@findex fftw_mpi_execute_dft
+@findex fftw_mpi_execute_dft_r2c
+@findex fftw_mpi_execute_dft_c2r
+@findex fftw_mpi_execute_r2r
+@example
+void fftw_mpi_execute_dft(fftw_plan p, fftw_complex *in, fftw_complex *out);
+void fftw_mpi_execute_dft_r2c(fftw_plan p, double *in, fftw_complex *out);
+void fftw_mpi_execute_dft_c2r(fftw_plan p, fftw_complex *in, double *out);
+void fftw_mpi_execute_r2r(fftw_plan p, double *in, double *out);
+@end example
+
+@cindex alignment
+@findex fftw_malloc
+These functions have the same restrictions as those of the serial
+new-array execute functions.  They are @emph{always} safe to apply to
+the @emph{same} @code{in} and @code{out} arrays that were used to
+create the plan.  They can only be applied to new arrarys if those
+arrays have the same types, dimensions, in-placeness, and alignment as
+the original arrays, where the best way to ensure the same alignment
+is to use FFTW's @code{fftw_malloc} and related allocation functions
+for all arrays (@pxref{Memory Allocation}).  Note that distributed
+transposes (@pxref{FFTW MPI Transposes}) use
+@code{fftw_mpi_execute_r2r}, since they count as rank-zero r2r plans
+from FFTW's perspective.
+
+@node MPI Data Distribution Functions, MPI Plan Creation, Using MPI Plans, FFTW MPI Reference
+@subsection MPI Data Distribution Functions
+
+@cindex data distribution
+As described above (@pxref{MPI Data Distribution}), in order to
+allocate your arrays, @emph{before} creating a plan, you must first
+call one of the following routines to determine the required
+allocation size and the portion of the array locally stored on a given
+process.  The @code{MPI_Comm} communicator passed here must be
+equivalent to the communicator used below for plan creation.
+
+The basic interface for multidimensional transforms consists of the
+functions:
+
+@findex fftw_mpi_local_size_2d
+@findex fftw_mpi_local_size_3d
+@findex fftw_mpi_local_size
+@findex fftw_mpi_local_size_2d_transposed
+@findex fftw_mpi_local_size_3d_transposed
+@findex fftw_mpi_local_size_transposed
+@example
+ptrdiff_t fftw_mpi_local_size_2d(ptrdiff_t n0, ptrdiff_t n1, MPI_Comm comm,
+                                 ptrdiff_t *local_n0, ptrdiff_t *local_0_start);
+ptrdiff_t fftw_mpi_local_size_3d(ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2,
+                                 MPI_Comm comm,
+                                 ptrdiff_t *local_n0, ptrdiff_t *local_0_start);
+ptrdiff_t fftw_mpi_local_size(int rnk, const ptrdiff_t *n, MPI_Comm comm,
+                              ptrdiff_t *local_n0, ptrdiff_t *local_0_start);
+
+ptrdiff_t fftw_mpi_local_size_2d_transposed(ptrdiff_t n0, ptrdiff_t n1, MPI_Comm comm,
+                                            ptrdiff_t *local_n0, ptrdiff_t *local_0_start,
+                                            ptrdiff_t *local_n1, ptrdiff_t *local_1_start);
+ptrdiff_t fftw_mpi_local_size_3d_transposed(ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2,
+                                            MPI_Comm comm,
+                                            ptrdiff_t *local_n0, ptrdiff_t *local_0_start,
+                                            ptrdiff_t *local_n1, ptrdiff_t *local_1_start);
+ptrdiff_t fftw_mpi_local_size_transposed(int rnk, const ptrdiff_t *n, MPI_Comm comm,
+                                         ptrdiff_t *local_n0, ptrdiff_t *local_0_start,
+                                         ptrdiff_t *local_n1, ptrdiff_t *local_1_start);
+@end example
+
+These functions return the number of elements to allocate (complex
+numbers for DFT/r2c/c2r plans, real numbers for r2r plans), whereas
+the @code{local_n0} and @code{local_0_start} return the portion
+(@code{local_0_start} to @code{local_0_start + local_n0 - 1}) of the
+first dimension of an @ndims{} array that is stored on the local
+process.  @xref{Basic and advanced distribution interfaces}.  For
+@code{FFTW_MPI_TRANSPOSED_OUT} plans, the @samp{_transposed} variants
+are useful in order to also return the local portion of the first
+dimension in the @ndimstrans{} transposed output.  
+@xref{Transposed distributions}.  
+The advanced interface for multidimensional transforms is:
+
+@cindex advanced interface
+@findex fftw_mpi_local_size_many
+@findex fftw_mpi_local_size_many_transposed
+@example
+ptrdiff_t fftw_mpi_local_size_many(int rnk, const ptrdiff_t *n, ptrdiff_t howmany,
+                                   ptrdiff_t block0, MPI_Comm comm,
+                                   ptrdiff_t *local_n0, ptrdiff_t *local_0_start);
+ptrdiff_t fftw_mpi_local_size_many_transposed(int rnk, const ptrdiff_t *n, ptrdiff_t howmany,
+                                              ptrdiff_t block0, ptrdiff_t block1, MPI_Comm comm,
+                                              ptrdiff_t *local_n0, ptrdiff_t *local_0_start,
+                                              ptrdiff_t *local_n1, ptrdiff_t *local_1_start);
+@end example
+
+These differ from the basic interface in only two ways.  First, they
+allow you to specify block sizes @code{block0} and @code{block1} (the
+latter for the transposed output); you can pass
+@code{FFTW_MPI_DEFAULT_BLOCK} to use FFTW's default block size as in
+the basic interface.  Second, you can pass a @code{howmany} parameter,
+corresponding to the advanced planning interface below: this is for
+transforms of contiguous @code{howmany}-tuples of numbers
+(@code{howmany = 1} in the basic interface).
+
+The corresponding basic and advanced routines for one-dimensional
+transforms (currently only complex DFTs) are:
+
+@findex fftw_mpi_local_size_1d
+@findex fftw_mpi_local_size_many_1d
+@example
+ptrdiff_t fftw_mpi_local_size_1d(
+             ptrdiff_t n0, MPI_Comm comm, int sign, unsigned flags,
+             ptrdiff_t *local_ni, ptrdiff_t *local_i_start,
+             ptrdiff_t *local_no, ptrdiff_t *local_o_start);
+ptrdiff_t fftw_mpi_local_size_many_1d(
+             ptrdiff_t n0, ptrdiff_t howmany,
+             MPI_Comm comm, int sign, unsigned flags,
+             ptrdiff_t *local_ni, ptrdiff_t *local_i_start,
+             ptrdiff_t *local_no, ptrdiff_t *local_o_start);
+@end example
+
+@ctindex FFTW_MPI_SCRAMBLED_OUT
+@ctindex FFTW_MPI_SCRAMBLED_IN
+As above, the return value is the number of elements to allocate
+(complex numbers, for complex DFTs).  The @code{local_ni} and
+@code{local_i_start} arguments return the portion
+(@code{local_i_start} to @code{local_i_start + local_ni - 1}) of the
+1d array that is stored on this process for the transform
+@emph{input}, and @code{local_no} and @code{local_o_start} are the
+corresponding quantities for the input.  The @code{sign}
+(@code{FFTW_FORWARD} or @code{FFTW_BACKWARD}) and @code{flags} must
+match the arguments passed when creating a plan.  Although the inputs
+and outputs have different data distributions in general, it is
+guaranteed that the @emph{output} data distribution of an
+@code{FFTW_FORWARD} plan will match the @emph{input} data distribution
+of an @code{FFTW_BACKWARD} plan and vice versa; similarly for the
+@code{FFTW_MPI_SCRAMBLED_OUT} and @code{FFTW_MPI_SCRAMBLED_IN} flags.
+@xref{One-dimensional distributions}.
+
+@node MPI Plan Creation, MPI Wisdom Communication, MPI Data Distribution Functions, FFTW MPI Reference
+@subsection MPI Plan Creation
+
+@subsubheading Complex-data MPI DFTs
+
+Plans for complex-data DFTs (@pxref{2d MPI example}) are created by:
+
+@findex fftw_mpi_plan_dft_1d
+@findex fftw_mpi_plan_dft_2d
+@findex fftw_mpi_plan_dft_3d
+@findex fftw_mpi_plan_dft
+@findex fftw_mpi_plan_many_dft
+@example
+fftw_plan fftw_mpi_plan_dft_1d(ptrdiff_t n0, fftw_complex *in, fftw_complex *out,
+                               MPI_Comm comm, int sign, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_2d(ptrdiff_t n0, ptrdiff_t n1,
+                               fftw_complex *in, fftw_complex *out,
+                               MPI_Comm comm, int sign, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_3d(ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2,
+                               fftw_complex *in, fftw_complex *out,
+                               MPI_Comm comm, int sign, unsigned flags);
+fftw_plan fftw_mpi_plan_dft(int rnk, const ptrdiff_t *n, 
+                            fftw_complex *in, fftw_complex *out,
+                            MPI_Comm comm, int sign, unsigned flags);
+fftw_plan fftw_mpi_plan_many_dft(int rnk, const ptrdiff_t *n,
+                                 ptrdiff_t howmany, ptrdiff_t block, ptrdiff_t tblock,
+                                 fftw_complex *in, fftw_complex *out,
+                                 MPI_Comm comm, int sign, unsigned flags);
+@end example
+
+@cindex MPI communicator
+@cindex collective function
+These are similar to their serial counterparts (@pxref{Complex DFTs})
+in specifying the dimensions, sign, and flags of the transform.  The
+@code{comm} argument gives an MPI communicator that specifies the set
+of processes to participate in the transform; plan creation is a
+collective function that must be called for all processes in the
+communicator.  The @code{in} and @code{out} pointers refer only to a
+portion of the overall transform data (@pxref{MPI Data Distribution})
+as specified by the @samp{local_size} functions in the previous
+section.  Unless @code{flags} contains @code{FFTW_ESTIMATE}, these
+arrays are overwritten during plan creation as for the serial
+interface.  For multi-dimensional transforms, any dimensions @code{>
+1} are supported; for one-dimensional transforms, only composite
+(non-prime) @code{n0} are currently supported (unlike the serial
+FFTW).  Requesting an unsupported transform size will yield a
+@code{NULL} plan.  (As in the serial interface, highly composite sizes
+generally yield the best performance.)
+
+@cindex advanced interface
+@ctindex FFTW_MPI_DEFAULT_BLOCK
+@cindex stride
+The advanced-interface @code{fftw_mpi_plan_many_dft} additionally
+allows you to specify the block sizes for the first dimension
+(@code{block}) of the @ndims{} input data and the first dimension
+(@code{tblock}) of the @ndimstrans{} transposed data (at intermediate
+steps of the transform, and for the output if
+@code{FFTW_TRANSPOSED_OUT} is specified in @code{flags}).  These must
+be the same block sizes as were passed to the corresponding
+@samp{local_size} function; you can pass @code{FFTW_MPI_DEFAULT_BLOCK}
+to use FFTW's default block size as in the basic interface.  Also, the
+@code{howmany} parameter specifies that the transform is of contiguous
+@code{howmany}-tuples rather than individual complex numbers; this
+corresponds to the same parameter in the serial advanced interface
+(@pxref{Advanced Complex DFTs}) with @code{stride = howmany} and
+@code{dist = 1}.
+
+@subsubheading MPI flags
+
+The @code{flags} can be any of those for the serial FFTW
+(@pxref{Planner Flags}), and in addition may include one or more of
+the following MPI-specific flags, which improve performance at the
+cost of changing the output or input data formats.
+
+@itemize @bullet
+
+@item
+@ctindex FFTW_MPI_SCRAMBLED_OUT
+@ctindex FFTW_MPI_SCRAMBLED_IN
+@code{FFTW_MPI_SCRAMBLED_OUT}, @code{FFTW_MPI_SCRAMBLED_IN}: valid for
+1d transforms only, these flags indicate that the output/input of the
+transform are in an undocumented ``scrambled'' order.  A forward
+@code{FFTW_MPI_SCRAMBLED_OUT} transform can be inverted by a backward
+@code{FFTW_MPI_SCRAMBLED_IN} (times the usual 1/@i{N} normalization).
+@xref{One-dimensional distributions}.
+
+@item
+@ctindex FFTW_MPI_TRANSPOSED_OUT
+@ctindex FFTW_MPI_TRANSPOSED_IN
+@code{FFTW_MPI_TRANSPOSED_OUT}, @code{FFTW_MPI_TRANSPOSED_IN}: valid
+for multidimensional (@code{rnk > 1}) transforms only, these flags
+specify that the output or input of an @ndims{} transform is
+transposed to @ndimstrans{}.  @xref{Transposed distributions}.
+
+@end itemize
+
+@subsubheading Real-data MPI DFTs
+
+@cindex r2c
+Plans for real-input/output (r2c/c2r) DFTs (@pxref{Multi-dimensional
+MPI DFTs of Real Data}) are created by:
+
+@findex fftw_mpi_plan_dft_r2c_2d
+@findex fftw_mpi_plan_dft_r2c_2d
+@findex fftw_mpi_plan_dft_r2c_3d
+@findex fftw_mpi_plan_dft_r2c
+@findex fftw_mpi_plan_dft_c2r_2d
+@findex fftw_mpi_plan_dft_c2r_2d
+@findex fftw_mpi_plan_dft_c2r_3d
+@findex fftw_mpi_plan_dft_c2r
+@example
+fftw_plan fftw_mpi_plan_dft_r2c_2d(ptrdiff_t n0, ptrdiff_t n1, 
+                                   double *in, fftw_complex *out,
+                                   MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_r2c_2d(ptrdiff_t n0, ptrdiff_t n1, 
+                                   double *in, fftw_complex *out,
+                                   MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_r2c_3d(ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2,
+                                   double *in, fftw_complex *out,
+                                   MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_r2c(int rnk, const ptrdiff_t *n,
+                                double *in, fftw_complex *out,
+                                MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_c2r_2d(ptrdiff_t n0, ptrdiff_t n1, 
+                                   fftw_complex *in, double *out,
+                                   MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_c2r_2d(ptrdiff_t n0, ptrdiff_t n1, 
+                                   fftw_complex *in, double *out,
+                                   MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_c2r_3d(ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2,
+                                   fftw_complex *in, double *out,
+                                   MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_dft_c2r(int rnk, const ptrdiff_t *n,
+                                fftw_complex *in, double *out,
+                                MPI_Comm comm, unsigned flags);
+@end example
+
+Similar to the serial interface (@pxref{Real-data DFTs}), these
+transform logically @ndims{} real data to/from @ndimshalf{} complex
+data, representing the non-redundant half of the conjugate-symmetry
+output of a real-input DFT (@pxref{Multi-dimensional Transforms}).
+However, the real array must be stored within a padded @ndimspad{}
+array (much like the in-place serial r2c transforms, but here for
+out-of-place transforms as well). Currently, only multi-dimensional
+(@code{rnk > 1}) r2c/c2r transforms are supported (requesting a plan
+for @code{rnk = 1} will yield @code{NULL}).  As explained above
+(@pxref{Multi-dimensional MPI DFTs of Real Data}), the data
+distribution of both the real and complex arrays is given by the
+@samp{local_size} function called for the dimensions of the
+@emph{complex} array.  Similar to the other planning functions, the
+input and output arrays are overwritten when the plan is created
+except in @code{FFTW_ESTIMATE} mode.
+
+As for the complex DFTs above, there is an advance interface that
+allows you to manually specify block sizes and to transform contiguous
+@code{howmany}-tuples of real/complex numbers:
+
+@findex fftw_mpi_plan_many_dft_r2c
+@findex fftw_mpi_plan_many_dft_c2r
+@example
+fftw_plan fftw_mpi_plan_many_dft_r2c
+              (int rnk, const ptrdiff_t *n, ptrdiff_t howmany,
+               ptrdiff_t iblock, ptrdiff_t oblock,
+               double *in, fftw_complex *out,
+               MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_many_dft_c2r
+              (int rnk, const ptrdiff_t *n, ptrdiff_t howmany,
+               ptrdiff_t iblock, ptrdiff_t oblock,
+               fftw_complex *in, double *out,
+               MPI_Comm comm, unsigned flags);               
+@end example
+
+@subsubheading MPI r2r transforms
+
+@cindex r2r
+There are corresponding plan-creation routines for r2r
+transforms (@pxref{More DFTs of Real Data}), currently supporting
+multidimensional (@code{rnk > 1}) transforms only (@code{rnk = 1} will
+yield a @code{NULL} plan):
+
+@example
+fftw_plan fftw_mpi_plan_r2r_2d(ptrdiff_t n0, ptrdiff_t n1,
+                               double *in, double *out,
+                               MPI_Comm comm,
+                               fftw_r2r_kind kind0, fftw_r2r_kind kind1,
+                               unsigned flags);
+fftw_plan fftw_mpi_plan_r2r_3d(ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2,
+                               double *in, double *out,
+                               MPI_Comm comm,
+                               fftw_r2r_kind kind0, fftw_r2r_kind kind1, fftw_r2r_kind kind2,
+                               unsigned flags);
+fftw_plan fftw_mpi_plan_r2r(int rnk, const ptrdiff_t *n,
+                            double *in, double *out,
+                            MPI_Comm comm, const fftw_r2r_kind *kind, 
+                            unsigned flags);
+fftw_plan fftw_mpi_plan_many_r2r(int rnk, const ptrdiff_t *n,
+                                 ptrdiff_t iblock, ptrdiff_t oblock,
+                                 double *in, double *out,
+                                 MPI_Comm comm, const fftw_r2r_kind *kind, 
+                                 unsigned flags);
+@end example
+
+The parameters are much the same as for the complex DFTs above, except
+that the arrays are of real numbers (and hence the outputs of the
+@samp{local_size} data-distribution functions should be interpreted as
+counts of real rather than complex numbers).  Also, the @code{kind}
+parameters specify the r2r kinds along each dimension as for the
+serial interface (@pxref{Real-to-Real Transform Kinds}).  @xref{Other
+Multi-dimensional Real-data MPI Transforms}.
+
+@subsubheading MPI transposition
+@cindex transpose
+
+FFTW also provides routines to plan a transpose of a distributed
+@code{n0} by @code{n1} array of real numbers, or an array of
+@code{howmany}-tuples of real numbers with specified block sizes
+(@pxref{FFTW MPI Transposes}):
+
+@findex fftw_mpi_plan_transpose
+@findex fftw_mpi_plan_many_transpose
+@example
+fftw_plan fftw_mpi_plan_transpose(ptrdiff_t n0, ptrdiff_t n1,
+                                  double *in, double *out,
+                                  MPI_Comm comm, unsigned flags);
+fftw_plan fftw_mpi_plan_many_transpose
+                (ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t howmany,
+                 ptrdiff_t block0, ptrdiff_t block1,
+                 double *in, double *out, MPI_Comm comm, unsigned flags);
+@end example
+
+@cindex new-array execution
+@findex fftw_mpi_execute_r2r
+These plans are used with the @code{fftw_mpi_execute_r2r} new-array
+execute function (@pxref{Using MPI Plans }), since they count as (rank
+zero) r2r plans from FFTW's perspective.
+
+@node MPI Wisdom Communication,  , MPI Plan Creation, FFTW MPI Reference
+@subsection MPI Wisdom Communication
+
+To facilitate synchronizing wisdom among the different MPI processes,
+we provide two functions:
+
+@findex fftw_mpi_gather_wisdom
+@findex fftw_mpi_broadcast_wisdom
+@example
+void fftw_mpi_gather_wisdom(MPI_Comm comm);
+void fftw_mpi_broadcast_wisdom(MPI_Comm comm);
+@end example
+
+The @code{fftw_mpi_gather_wisdom} function gathers all wisdom in the
+given communicator @code{comm} to the process of rank 0 in the
+communicator: that process obtains the union of all wisdom on all the
+processes.  As a side effect, some other processes will gain
+additional wisdom from other processes, but only process 0 will gain
+the complete union.
+
+The @code{fftw_mpi_broadcast_wisdom} does the reverse: it exports
+wisdom from process 0 in @code{comm} to all other processes in the
+communicator, replacing any wisdom they currently have.
+
+@xref{FFTW MPI Wisdom}.
+
+@c ------------------------------------------------------------
+@node FFTW MPI Fortran Interface,  , FFTW MPI Reference, Distributed-memory FFTW with MPI
+@section FFTW MPI Fortran Interface
+@cindex Fortran interface
+
+@cindex iso_c_binding
+The FFTW MPI interface is callable from modern Fortran compilers
+supporting the Fortran 2003 @code{iso_c_binding} standard for calling
+C functions.  As described in @ref{Calling FFTW from Modern Fortran},
+this means that you can directly call FFTW's C interface from Fortran
+with only minor changes in syntax.  There are, however, a few things
+specific to the MPI interface to keep in mind:
+
+@itemize @bullet
+
+@item
+Instead of including @code{fftw3.f03} as in @ref{Overview of Fortran
+interface }, you should @code{include 'fftw3-mpi.f03'} (after
+@code{use, intrinsic :: iso_c_binding} as before).  The
+@code{fftw3-mpi.f03} file includes @code{fftw3.f03}, so you should
+@emph{not} @code{include} them both yourself.  (You will also want to
+include the MPI header file, usually via @code{include 'mpif.h'} or
+similar, although though this is not needed by @code{fftw3-mpi.f03}
+@i{per se}.)  (To use the @samp{fftwl_} @code{long double} extended-precision routines in supporting compilers, you should include @code{fftw3f-mpi.f03} in @emph{addition} to @code{fftw3-mpi.f03}. @xref{Extended and quadruple precision in Fortran}.)
+
+@item
+Because of the different storage conventions between C and Fortran,
+you reverse the order of your array dimensions when passing them to
+FFTW (@pxref{Reversing array dimensions}).  This is merely a
+difference in notation and incurs no performance overhead.  However,
+it means that, whereas in C the @emph{first} dimension is distributed,
+in Fortran the @emph{last} dimension of your array is distributed.
+
+@item
+@cindex MPI communicator
+In Fortran, communicators are stored as @code{integer} types; there is
+no @code{MPI_Comm} type, nor is there any way to access a C
+@code{MPI_Comm}.  Fortunately, this is taken care of for you by the
+FFTW Fortran interface: whenever the C interface expects an
+@code{MPI_Comm} type, you should pass the Fortran communicator as an
+@code{integer}.@footnote{Technically, this is because you aren't
+actually calling the C functions directly. You are calling wrapper
+functions that translate the communicator with @code{MPI_Comm_f2c}
+before calling the ordinary C interface.  This is all done
+transparently, however, since the @code{fftw3-mpi.f03} interface file
+renames the wrappers so that they are called in Fortran with the same
+names as the C interface functions.}
+
+@item
+Because you need to call the @samp{local_size} function to find out
+how much space to allocate, and this may be @emph{larger} than the
+local portion of the array (@pxref{MPI Data Distribution}), you should
+@emph{always} allocate your arrays dynamically using FFTW's allocation
+routines as described in @ref{Allocating aligned memory in Fortran}.
+(Coincidentally, this also provides the best performance by
+guaranteeding proper data alignment.)
+
+@item
+Because all sizes in the MPI FFTW interface are declared as
+@code{ptrdiff_t} in C, you should use @code{integer(C_INTPTR_T)} in
+Fortran (@pxref{FFTW Fortran type reference}).
+
+@item
+@findex fftw_execute_dft
+@findex fftw_mpi_execute_dft
+@cindex new-array execution
+In Fortran, because of the language semantics, we generally recommend
+using the new-array execute functions for all plans, even in the
+common case where you are executing the plan on the same arrays for
+which the plan was created (@pxref{Plan execution in Fortran}).
+However, note that in the MPI interface these functions are changed:
+@code{fftw_execute_dft} becomes @code{fftw_mpi_execute_dft},
+etcetera. @xref{Using MPI Plans}.
+
+@end itemize
+
+For example, here is a Fortran code snippet to perform a distributed
+@twodims{L,M} complex DFT in-place.  (This assumes you have already
+initialized MPI with @code{MPI_init} and have also performed
+@code{call fftw_mpi_init}.)
+
+@example
+  use, intrinsic :: iso_c_binding
+  include 'fftw3-mpi.f03'
+  integer(C_INTPTR_T), parameter :: L = ...
+  integer(C_INTPTR_T), parameter :: M = ...
+  type(C_PTR) :: plan, cdata
+  complex(C_DOUBLE_COMPLEX), pointer :: data(:,:)
+  integer(C_INTPTR_T) :: i, j, alloc_local, local_M, local_j_offset
+
+!   @r{get local data size and allocate (note dimension reversal)}
+  alloc_local = fftw_mpi_local_size_2d(M, L, MPI_COMM_WORLD, &
+                                       local_M, local_j_offset)
+  cdata = fftw_alloc_complex(alloc_local)
+  call c_f_pointer(cdata, data, [L,local_M])
+
+!   @r{create MPI plan for in-place forward DFT (note dimension reversal)}
+  plan = fftw_mpi_plan_dft_2d(M, L, data, data, MPI_COMM_WORLD, &
+                              FFTW_FORWARD, FFTW_MEASURE)
+
+! @r{initialize data to some function} my_function(i,j)
+  do j = 1, local_M
+    do i = 1, L
+      data(i, j) = my_function(i, j + local_j_offset)
+    end do
+  end do
+
+! @r{compute transform (as many times as desired)}
+  call fftw_mpi_execute_dft(plan, data, data)
+
+  call fftw_destroy_plan(plan)
+  call fftw_free(cdata)
+@end example
+
+Note that when we called @code{fftw_mpi_local_size_2d} and
+@code{fftw_mpi_plan_dft_2d} with the dimensions in reversed order,
+since a @twodims{L,M} Fortran array is viewed by FFTW in C as a
+@twodims{M, L} array.  This means that the array was distributed over
+the @code{M} dimension, the local portion of which is a
+@twodims{L,local_M} array in Fortran.  (You must @emph{not} use an
+@code{allocate} statement to allocate an @twodims{L,local_M} array,
+however; you must allocate @code{alloc_local} complex numbers, which
+may be greater than @code{L * local_M}, in order to reserve space for
+intermediate steps of the transform.)  Finally, we mention that
+because C's array indices are zero-based, the @code{local_j_offset}
+argument can conveniently be interpreted as an offset in the 1-based
+@code{j} index (rather than as a starting index as in C).
+
+If instead you had used the @code{ior(FFTW_MEASURE,
+FFTW_MPI_TRANSPOSED_OUT)} flag, the output of the transform would be a
+transposed @twodims{M,local_L} array, associated with the @emph{same}
+@code{cdata} allocation (since the transform is in-place), and which
+you could declare with:
+
+@example
+  complex(C_DOUBLE_COMPLEX), pointer :: tdata(:,:)
+  ...
+  call c_f_pointer(cdata, tdata, [M,local_L])
+@end example
+
+where @code{local_L} would have been obtained by changing the
+@code{fftw_mpi_local_size_2d} call to:
+
+@example
+  alloc_local = fftw_mpi_local_size_2d_transposed(M, L, MPI_COMM_WORLD, &
+                           local_M, local_j_offset, local_L, local_i_offset)
+@end example