|
cannam@167
|
1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
|
cannam@167
|
2 <html>
|
|
cannam@167
|
3 <!-- This manual is for FFTW
|
|
cannam@167
|
4 (version 3.3.8, 24 May 2018).
|
|
cannam@167
|
5
|
|
cannam@167
|
6 Copyright (C) 2003 Matteo Frigo.
|
|
cannam@167
|
7
|
|
cannam@167
|
8 Copyright (C) 2003 Massachusetts Institute of Technology.
|
|
cannam@167
|
9
|
|
cannam@167
|
10 Permission is granted to make and distribute verbatim copies of this
|
|
cannam@167
|
11 manual provided the copyright notice and this permission notice are
|
|
cannam@167
|
12 preserved on all copies.
|
|
cannam@167
|
13
|
|
cannam@167
|
14 Permission is granted to copy and distribute modified versions of this
|
|
cannam@167
|
15 manual under the conditions for verbatim copying, provided that the
|
|
cannam@167
|
16 entire resulting derived work is distributed under the terms of a
|
|
cannam@167
|
17 permission notice identical to this one.
|
|
cannam@167
|
18
|
|
cannam@167
|
19 Permission is granted to copy and distribute translations of this manual
|
|
cannam@167
|
20 into another language, under the above conditions for modified versions,
|
|
cannam@167
|
21 except that this permission notice may be stated in a translation
|
|
cannam@167
|
22 approved by the Free Software Foundation. -->
|
|
cannam@167
|
23 <!-- Created by GNU Texinfo 6.3, http://www.gnu.org/software/texinfo/ -->
|
|
cannam@167
|
24 <head>
|
|
cannam@167
|
25 <title>FFTW 3.3.8: MPI Data Distribution</title>
|
|
cannam@167
|
26
|
|
cannam@167
|
27 <meta name="description" content="FFTW 3.3.8: MPI Data Distribution">
|
|
cannam@167
|
28 <meta name="keywords" content="FFTW 3.3.8: MPI Data Distribution">
|
|
cannam@167
|
29 <meta name="resource-type" content="document">
|
|
cannam@167
|
30 <meta name="distribution" content="global">
|
|
cannam@167
|
31 <meta name="Generator" content="makeinfo">
|
|
cannam@167
|
32 <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
|
|
cannam@167
|
33 <link href="index.html#Top" rel="start" title="Top">
|
|
cannam@167
|
34 <link href="Concept-Index.html#Concept-Index" rel="index" title="Concept Index">
|
|
cannam@167
|
35 <link href="index.html#SEC_Contents" rel="contents" title="Table of Contents">
|
|
cannam@167
|
36 <link href="Distributed_002dmemory-FFTW-with-MPI.html#Distributed_002dmemory-FFTW-with-MPI" rel="up" title="Distributed-memory FFTW with MPI">
|
|
cannam@167
|
37 <link href="Basic-and-advanced-distribution-interfaces.html#Basic-and-advanced-distribution-interfaces" rel="next" title="Basic and advanced distribution interfaces">
|
|
cannam@167
|
38 <link href="2d-MPI-example.html#g_t2d-MPI-example" rel="prev" title="2d MPI example">
|
|
cannam@167
|
39 <style type="text/css">
|
|
cannam@167
|
40 <!--
|
|
cannam@167
|
41 a.summary-letter {text-decoration: none}
|
|
cannam@167
|
42 blockquote.indentedblock {margin-right: 0em}
|
|
cannam@167
|
43 blockquote.smallindentedblock {margin-right: 0em; font-size: smaller}
|
|
cannam@167
|
44 blockquote.smallquotation {font-size: smaller}
|
|
cannam@167
|
45 div.display {margin-left: 3.2em}
|
|
cannam@167
|
46 div.example {margin-left: 3.2em}
|
|
cannam@167
|
47 div.lisp {margin-left: 3.2em}
|
|
cannam@167
|
48 div.smalldisplay {margin-left: 3.2em}
|
|
cannam@167
|
49 div.smallexample {margin-left: 3.2em}
|
|
cannam@167
|
50 div.smalllisp {margin-left: 3.2em}
|
|
cannam@167
|
51 kbd {font-style: oblique}
|
|
cannam@167
|
52 pre.display {font-family: inherit}
|
|
cannam@167
|
53 pre.format {font-family: inherit}
|
|
cannam@167
|
54 pre.menu-comment {font-family: serif}
|
|
cannam@167
|
55 pre.menu-preformatted {font-family: serif}
|
|
cannam@167
|
56 pre.smalldisplay {font-family: inherit; font-size: smaller}
|
|
cannam@167
|
57 pre.smallexample {font-size: smaller}
|
|
cannam@167
|
58 pre.smallformat {font-family: inherit; font-size: smaller}
|
|
cannam@167
|
59 pre.smalllisp {font-size: smaller}
|
|
cannam@167
|
60 span.nolinebreak {white-space: nowrap}
|
|
cannam@167
|
61 span.roman {font-family: initial; font-weight: normal}
|
|
cannam@167
|
62 span.sansserif {font-family: sans-serif; font-weight: normal}
|
|
cannam@167
|
63 ul.no-bullet {list-style: none}
|
|
cannam@167
|
64 -->
|
|
cannam@167
|
65 </style>
|
|
cannam@167
|
66
|
|
cannam@167
|
67
|
|
cannam@167
|
68 </head>
|
|
cannam@167
|
69
|
|
cannam@167
|
70 <body lang="en">
|
|
cannam@167
|
71 <a name="MPI-Data-Distribution"></a>
|
|
cannam@167
|
72 <div class="header">
|
|
cannam@167
|
73 <p>
|
|
cannam@167
|
74 Next: <a href="Multi_002ddimensional-MPI-DFTs-of-Real-Data.html#Multi_002ddimensional-MPI-DFTs-of-Real-Data" accesskey="n" rel="next">Multi-dimensional MPI DFTs of Real Data</a>, Previous: <a href="2d-MPI-example.html#g_t2d-MPI-example" accesskey="p" rel="prev">2d MPI example</a>, Up: <a href="Distributed_002dmemory-FFTW-with-MPI.html#Distributed_002dmemory-FFTW-with-MPI" accesskey="u" rel="up">Distributed-memory FFTW with MPI</a> [<a href="index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
|
|
cannam@167
|
75 </div>
|
|
cannam@167
|
76 <hr>
|
|
cannam@167
|
77 <a name="MPI-Data-Distribution-1"></a>
|
|
cannam@167
|
78 <h3 class="section">6.4 MPI Data Distribution</h3>
|
|
cannam@167
|
79 <a name="index-data-distribution-2"></a>
|
|
cannam@167
|
80
|
|
cannam@167
|
81 <p>The most important concept to understand in using FFTW’s MPI interface
|
|
cannam@167
|
82 is the data distribution. With a serial or multithreaded FFT, all of
|
|
cannam@167
|
83 the inputs and outputs are stored as a single contiguous chunk of
|
|
cannam@167
|
84 memory. With a distributed-memory FFT, the inputs and outputs are
|
|
cannam@167
|
85 broken into disjoint blocks, one per process.
|
|
cannam@167
|
86 </p>
|
|
cannam@167
|
87 <p>In particular, FFTW uses a <em>1d block distribution</em> of the data,
|
|
cannam@167
|
88 distributed along the <em>first dimension</em>. For example, if you
|
|
cannam@167
|
89 want to perform a 100 × 200
|
|
cannam@167
|
90 complex DFT, distributed over 4
|
|
cannam@167
|
91 processes, each process will get a 25 × 200
|
|
cannam@167
|
92 slice of the data.
|
|
cannam@167
|
93 That is, process 0 will get rows 0 through 24, process 1 will get rows
|
|
cannam@167
|
94 25 through 49, process 2 will get rows 50 through 74, and process 3
|
|
cannam@167
|
95 will get rows 75 through 99. If you take the same array but
|
|
cannam@167
|
96 distribute it over 3 processes, then it is not evenly divisible so the
|
|
cannam@167
|
97 different processes will have unequal chunks. FFTW’s default choice
|
|
cannam@167
|
98 in this case is to assign 34 rows to processes 0 and 1, and 32 rows to
|
|
cannam@167
|
99 process 2.
|
|
cannam@167
|
100 <a name="index-block-distribution"></a>
|
|
cannam@167
|
101 </p>
|
|
cannam@167
|
102
|
|
cannam@167
|
103 <p>FFTW provides several ‘<samp>fftw_mpi_local_size</samp>’ routines that you can
|
|
cannam@167
|
104 call to find out what portion of an array is stored on the current
|
|
cannam@167
|
105 process. In most cases, you should use the default block sizes picked
|
|
cannam@167
|
106 by FFTW, but it is also possible to specify your own block size. For
|
|
cannam@167
|
107 example, with a 100 × 200
|
|
cannam@167
|
108 array on three processes, you can
|
|
cannam@167
|
109 tell FFTW to use a block size of 40, which would assign 40 rows to
|
|
cannam@167
|
110 processes 0 and 1, and 20 rows to process 2. FFTW’s default is to
|
|
cannam@167
|
111 divide the data equally among the processes if possible, and as best
|
|
cannam@167
|
112 it can otherwise. The rows are always assigned in “rank order,”
|
|
cannam@167
|
113 i.e. process 0 gets the first block of rows, then process 1, and so
|
|
cannam@167
|
114 on. (You can change this by using <code>MPI_Comm_split</code> to create a
|
|
cannam@167
|
115 new communicator with re-ordered processes.) However, you should
|
|
cannam@167
|
116 always call the ‘<samp>fftw_mpi_local_size</samp>’ routines, if possible,
|
|
cannam@167
|
117 rather than trying to predict FFTW’s distribution choices.
|
|
cannam@167
|
118 </p>
|
|
cannam@167
|
119 <p>In particular, it is critical that you allocate the storage size that
|
|
cannam@167
|
120 is returned by ‘<samp>fftw_mpi_local_size</samp>’, which is <em>not</em>
|
|
cannam@167
|
121 necessarily the size of the local slice of the array. The reason is
|
|
cannam@167
|
122 that intermediate steps of FFTW’s algorithms involve transposing the
|
|
cannam@167
|
123 array and redistributing the data, so at these intermediate steps FFTW
|
|
cannam@167
|
124 may require more local storage space (albeit always proportional to
|
|
cannam@167
|
125 the total size divided by the number of processes). The
|
|
cannam@167
|
126 ‘<samp>fftw_mpi_local_size</samp>’ functions know how much storage is required
|
|
cannam@167
|
127 for these intermediate steps and tell you the correct amount to
|
|
cannam@167
|
128 allocate.
|
|
cannam@167
|
129 </p>
|
|
cannam@167
|
130 <table class="menu" border="0" cellspacing="0">
|
|
cannam@167
|
131 <tr><td align="left" valign="top">• <a href="Basic-and-advanced-distribution-interfaces.html#Basic-and-advanced-distribution-interfaces" accesskey="1">Basic and advanced distribution interfaces</a>:</td><td> </td><td align="left" valign="top">
|
|
cannam@167
|
132 </td></tr>
|
|
cannam@167
|
133 <tr><td align="left" valign="top">• <a href="Load-balancing.html#Load-balancing" accesskey="2">Load balancing</a>:</td><td> </td><td align="left" valign="top">
|
|
cannam@167
|
134 </td></tr>
|
|
cannam@167
|
135 <tr><td align="left" valign="top">• <a href="Transposed-distributions.html#Transposed-distributions" accesskey="3">Transposed distributions</a>:</td><td> </td><td align="left" valign="top">
|
|
cannam@167
|
136 </td></tr>
|
|
cannam@167
|
137 <tr><td align="left" valign="top">• <a href="One_002ddimensional-distributions.html#One_002ddimensional-distributions" accesskey="4">One-dimensional distributions</a>:</td><td> </td><td align="left" valign="top">
|
|
cannam@167
|
138 </td></tr>
|
|
cannam@167
|
139 </table>
|
|
cannam@167
|
140
|
|
cannam@167
|
141 <hr>
|
|
cannam@167
|
142 <div class="header">
|
|
cannam@167
|
143 <p>
|
|
cannam@167
|
144 Next: <a href="Multi_002ddimensional-MPI-DFTs-of-Real-Data.html#Multi_002ddimensional-MPI-DFTs-of-Real-Data" accesskey="n" rel="next">Multi-dimensional MPI DFTs of Real Data</a>, Previous: <a href="2d-MPI-example.html#g_t2d-MPI-example" accesskey="p" rel="prev">2d MPI example</a>, Up: <a href="Distributed_002dmemory-FFTW-with-MPI.html#Distributed_002dmemory-FFTW-with-MPI" accesskey="u" rel="up">Distributed-memory FFTW with MPI</a> [<a href="index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
|
|
cannam@167
|
145 </div>
|
|
cannam@167
|
146
|
|
cannam@167
|
147
|
|
cannam@167
|
148
|
|
cannam@167
|
149 </body>
|
|
cannam@167
|
150 </html>
|
