Chris@19: Chris@19:
Chris@19:Chris@19: Next: One-dimensional distributions, Chris@19: Previous: Load balancing, Chris@19: Up: MPI Data Distribution Chris@19:
Internally, FFTW's MPI transform algorithms work by first computing
Chris@19: transforms of the data local to each process, then by globally
Chris@19: transposing the data in some fashion to redistribute the data
Chris@19: among the processes, transforming the new data local to each process,
Chris@19: and transposing back. For example, a two-dimensional n0
by
Chris@19: n1
array, distributed across the n0
dimension, is
Chris@19: transformd by: (i) transforming the n1
dimension, which are
Chris@19: local to each process; (ii) transposing to an n1
by n0
Chris@19: array, distributed across the n1
dimension; (iii) transforming
Chris@19: the n0
dimension, which is now local to each process; (iv)
Chris@19: transposing back.
Chris@19:
Chris@19:
Chris@19:
However, in many applications it is acceptable to compute a
Chris@19: multidimensional DFT whose results are produced in transposed order
Chris@19: (e.g., n1
by n0
in two dimensions). This provides a
Chris@19: significant performance advantage, because it means that the final
Chris@19: transposition step can be omitted. FFTW supports this optimization,
Chris@19: which you specify by passing the flag FFTW_MPI_TRANSPOSED_OUT
Chris@19: to the planner routines. To compute the inverse transform of
Chris@19: transposed output, you specify FFTW_MPI_TRANSPOSED_IN
to tell
Chris@19: it that the input is transposed. In this section, we explain how to
Chris@19: interpret the output format of such a transform.
Chris@19:
Chris@19:
Chris@19:
Suppose you have are transforming multi-dimensional data with (at
Chris@19: least two) dimensions n0 × n1 × n2 × … × nd-1. As always, it is distributed along
Chris@19: the first dimension n0. Now, if we compute its DFT with the
Chris@19: FFTW_MPI_TRANSPOSED_OUT
flag, the resulting output data are stored
Chris@19: with the first two dimensions transposed: n1 × n0 × n2 ×…× nd-1,
Chris@19: distributed along the n1 dimension. Conversely, if we take the
Chris@19: n1 × n0 × n2 ×…× nd-1 data and transform it with the
Chris@19: FFTW_MPI_TRANSPOSED_IN
flag, then the format goes back to the
Chris@19: original n0 × n1 × n2 × … × nd-1 array.
Chris@19:
Chris@19:
There are two ways to find the portion of the transposed array that Chris@19: resides on the current process. First, you can simply call the Chris@19: appropriate ‘local_size’ function, passing n1 × n0 × n2 ×…× nd-1 (the Chris@19: transposed dimensions). This would mean calling the ‘local_size’ Chris@19: function twice, once for the transposed and once for the Chris@19: non-transposed dimensions. Alternatively, you can call one of the Chris@19: ‘local_size_transposed’ functions, which returns both the Chris@19: non-transposed and transposed data distribution from a single call. Chris@19: For example, for a 3d transform with transposed output (or input), you Chris@19: might call: Chris@19: Chris@19:
ptrdiff_t fftw_mpi_local_size_3d_transposed( Chris@19: ptrdiff_t n0, ptrdiff_t n1, ptrdiff_t n2, MPI_Comm comm, Chris@19: ptrdiff_t *local_n0, ptrdiff_t *local_0_start, Chris@19: ptrdiff_t *local_n1, ptrdiff_t *local_1_start); Chris@19:Chris@19:
Chris@19: Here, local_n0
and local_0_start
give the size and
Chris@19: starting index of the n0
dimension for the
Chris@19: non-transposed data, as in the previous sections. For
Chris@19: transposed data (e.g. the output for
Chris@19: FFTW_MPI_TRANSPOSED_OUT
), local_n1
and
Chris@19: local_1_start
give the size and starting index of the n1
Chris@19: dimension, which is the first dimension of the transposed data
Chris@19: (n1
by n0
by n2
).
Chris@19:
Chris@19:
(Note that FFTW_MPI_TRANSPOSED_IN
is completely equivalent to
Chris@19: performing FFTW_MPI_TRANSPOSED_OUT
and passing the first two
Chris@19: dimensions to the planner in reverse order, or vice versa. If you
Chris@19: pass both the FFTW_MPI_TRANSPOSED_IN
and
Chris@19: FFTW_MPI_TRANSPOSED_OUT
flags, it is equivalent to swapping the
Chris@19: first two dimensions passed to the planner and passing neither
Chris@19: flag.)
Chris@19:
Chris@19:
Chris@19: