Chris@42: Chris@42: Chris@42: Chris@42: Chris@42:
Chris@42:Chris@42: Next: MPI Data Distribution, Previous: Linking and Initializing MPI FFTW, Up: Distributed-memory FFTW with MPI [Contents][Index]
Chris@42:Before we document the FFTW MPI interface in detail, we begin with a
Chris@42: simple example outlining how one would perform a two-dimensional
Chris@42: N0
by N1
complex DFT.
Chris@42:
#include <fftw3-mpi.h> Chris@42: Chris@42: int main(int argc, char **argv) Chris@42: { Chris@42: const ptrdiff_t N0 = ..., N1 = ...; Chris@42: fftw_plan plan; Chris@42: fftw_complex *data; Chris@42: ptrdiff_t alloc_local, local_n0, local_0_start, i, j; Chris@42: Chris@42: MPI_Init(&argc, &argv); Chris@42: fftw_mpi_init(); Chris@42: Chris@42: /* get local data size and allocate */ Chris@42: alloc_local = fftw_mpi_local_size_2d(N0, N1, MPI_COMM_WORLD, Chris@42: &local_n0, &local_0_start); Chris@42: data = fftw_alloc_complex(alloc_local); Chris@42: Chris@42: /* create plan for in-place forward DFT */ Chris@42: plan = fftw_mpi_plan_dft_2d(N0, N1, data, data, MPI_COMM_WORLD, Chris@42: FFTW_FORWARD, FFTW_ESTIMATE); Chris@42: Chris@42: /* initialize data to some function my_function(x,y) */ Chris@42: for (i = 0; i < local_n0; ++i) for (j = 0; j < N1; ++j) Chris@42: data[i*N1 + j] = my_function(local_0_start + i, j); Chris@42: Chris@42: /* compute transforms, in-place, as many times as desired */ Chris@42: fftw_execute(plan); Chris@42: Chris@42: fftw_destroy_plan(plan); Chris@42: Chris@42: MPI_Finalize(); Chris@42: } Chris@42:
As can be seen above, the MPI interface follows the same basic style Chris@42: of allocate/plan/execute/destroy as the serial FFTW routines. All of Chris@42: the MPI-specific routines are prefixed with ‘fftw_mpi_’ instead Chris@42: of ‘fftw_’. There are a few important differences, however: Chris@42:
Chris@42:First, we must call fftw_mpi_init()
after calling
Chris@42: MPI_Init
(required in all MPI programs) and before calling any
Chris@42: other ‘fftw_mpi_’ routine.
Chris@42:
Chris@42:
Chris@42:
Second, when we create the plan with fftw_mpi_plan_dft_2d
,
Chris@42: analogous to fftw_plan_dft_2d
, we pass an additional argument:
Chris@42: the communicator, indicating which processes will participate in the
Chris@42: transform (here MPI_COMM_WORLD
, indicating all processes).
Chris@42: Whenever you create, execute, or destroy a plan for an MPI transform,
Chris@42: you must call the corresponding FFTW routine on all processes
Chris@42: in the communicator for that transform. (That is, these are
Chris@42: collective calls.) Note that the plan for the MPI transform
Chris@42: uses the standard fftw_execute
and fftw_destroy
routines
Chris@42: (on the other hand, there are MPI-specific new-array execute functions
Chris@42: documented below).
Chris@42:
Chris@42:
Chris@42:
Chris@42:
Third, all of the FFTW MPI routines take ptrdiff_t
arguments
Chris@42: instead of int
as for the serial FFTW. ptrdiff_t
is a
Chris@42: standard C integer type which is (at least) 32 bits wide on a 32-bit
Chris@42: machine and 64 bits wide on a 64-bit machine. This is to make it easy
Chris@42: to specify very large parallel transforms on a 64-bit machine. (You
Chris@42: can specify 64-bit transform sizes in the serial FFTW, too, but only
Chris@42: by using the ‘guru64’ planner interface. See 64-bit Guru Interface.)
Chris@42:
Chris@42:
Chris@42:
Fourth, and most importantly, you don’t allocate the entire
Chris@42: two-dimensional array on each process. Instead, you call
Chris@42: fftw_mpi_local_size_2d
to find out what portion of the
Chris@42: array resides on each processor, and how much space to allocate.
Chris@42: Here, the portion of the array on each process is a local_n0
by
Chris@42: N1
slice of the total array, starting at index
Chris@42: local_0_start
. The total number of fftw_complex
numbers
Chris@42: to allocate is given by the alloc_local
return value, which
Chris@42: may be greater than local_n0 * N1
(in case some
Chris@42: intermediate calculations require additional storage). The data
Chris@42: distribution in FFTW’s MPI interface is described in more detail by
Chris@42: the next section.
Chris@42:
Chris@42:
Chris@42:
Given the portion of the array that resides on the local process, it
Chris@42: is straightforward to initialize the data (here to a function
Chris@42: myfunction
) and otherwise manipulate it. Of course, at the end
Chris@42: of the program you may want to output the data somehow, but
Chris@42: synchronizing this output is up to you and is beyond the scope of this
Chris@42: manual. (One good way to output a large multi-dimensional distributed
Chris@42: array in MPI to a portable binary file is to use the free HDF5
Chris@42: library; see the HDF home page.)
Chris@42:
Chris@42:
Chris@42:
Chris@42: Next: MPI Data Distribution, Previous: Linking and Initializing MPI FFTW, Up: Distributed-memory FFTW with MPI [Contents][Index]
Chris@42: