|
cannam@95
|
1 <html lang="en">
|
|
cannam@95
|
2 <head>
|
|
cannam@95
|
3 <title>Combining MPI and Threads - FFTW 3.3.3</title>
|
|
cannam@95
|
4 <meta http-equiv="Content-Type" content="text/html">
|
|
cannam@95
|
5 <meta name="description" content="FFTW 3.3.3">
|
|
cannam@95
|
6 <meta name="generator" content="makeinfo 4.13">
|
|
cannam@95
|
7 <link title="Top" rel="start" href="index.html#Top">
|
|
cannam@95
|
8 <link rel="up" href="Distributed_002dmemory-FFTW-with-MPI.html#Distributed_002dmemory-FFTW-with-MPI" title="Distributed-memory FFTW with MPI">
|
|
cannam@95
|
9 <link rel="prev" href="FFTW-MPI-Performance-Tips.html#FFTW-MPI-Performance-Tips" title="FFTW MPI Performance Tips">
|
|
cannam@95
|
10 <link rel="next" href="FFTW-MPI-Reference.html#FFTW-MPI-Reference" title="FFTW MPI Reference">
|
|
cannam@95
|
11 <link href="http://www.gnu.org/software/texinfo/" rel="generator-home" title="Texinfo Homepage">
|
|
cannam@95
|
12 <!--
|
|
cannam@95
|
13 This manual is for FFTW
|
|
cannam@95
|
14 (version 3.3.3, 25 November 2012).
|
|
cannam@95
|
15
|
|
cannam@95
|
16 Copyright (C) 2003 Matteo Frigo.
|
|
cannam@95
|
17
|
|
cannam@95
|
18 Copyright (C) 2003 Massachusetts Institute of Technology.
|
|
cannam@95
|
19
|
|
cannam@95
|
20 Permission is granted to make and distribute verbatim copies of
|
|
cannam@95
|
21 this manual provided the copyright notice and this permission
|
|
cannam@95
|
22 notice are preserved on all copies.
|
|
cannam@95
|
23
|
|
cannam@95
|
24 Permission is granted to copy and distribute modified versions of
|
|
cannam@95
|
25 this manual under the conditions for verbatim copying, provided
|
|
cannam@95
|
26 that the entire resulting derived work is distributed under the
|
|
cannam@95
|
27 terms of a permission notice identical to this one.
|
|
cannam@95
|
28
|
|
cannam@95
|
29 Permission is granted to copy and distribute translations of this
|
|
cannam@95
|
30 manual into another language, under the above conditions for
|
|
cannam@95
|
31 modified versions, except that this permission notice may be
|
|
cannam@95
|
32 stated in a translation approved by the Free Software Foundation.
|
|
cannam@95
|
33 -->
|
|
cannam@95
|
34 <meta http-equiv="Content-Style-Type" content="text/css">
|
|
cannam@95
|
35 <style type="text/css"><!--
|
|
cannam@95
|
36 pre.display { font-family:inherit }
|
|
cannam@95
|
37 pre.format { font-family:inherit }
|
|
cannam@95
|
38 pre.smalldisplay { font-family:inherit; font-size:smaller }
|
|
cannam@95
|
39 pre.smallformat { font-family:inherit; font-size:smaller }
|
|
cannam@95
|
40 pre.smallexample { font-size:smaller }
|
|
cannam@95
|
41 pre.smalllisp { font-size:smaller }
|
|
cannam@95
|
42 span.sc { font-variant:small-caps }
|
|
cannam@95
|
43 span.roman { font-family:serif; font-weight:normal; }
|
|
cannam@95
|
44 span.sansserif { font-family:sans-serif; font-weight:normal; }
|
|
cannam@95
|
45 --></style>
|
|
cannam@95
|
46 </head>
|
|
cannam@95
|
47 <body>
|
|
cannam@95
|
48 <div class="node">
|
|
cannam@95
|
49 <a name="Combining-MPI-and-Threads"></a>
|
|
cannam@95
|
50 <p>
|
|
cannam@95
|
51 Next: <a rel="next" accesskey="n" href="FFTW-MPI-Reference.html#FFTW-MPI-Reference">FFTW MPI Reference</a>,
|
|
cannam@95
|
52 Previous: <a rel="previous" accesskey="p" href="FFTW-MPI-Performance-Tips.html#FFTW-MPI-Performance-Tips">FFTW MPI Performance Tips</a>,
|
|
cannam@95
|
53 Up: <a rel="up" accesskey="u" href="Distributed_002dmemory-FFTW-with-MPI.html#Distributed_002dmemory-FFTW-with-MPI">Distributed-memory FFTW with MPI</a>
|
|
cannam@95
|
54 <hr>
|
|
cannam@95
|
55 </div>
|
|
cannam@95
|
56
|
|
cannam@95
|
57 <h3 class="section">6.11 Combining MPI and Threads</h3>
|
|
cannam@95
|
58
|
|
cannam@95
|
59 <p><a name="index-threads-427"></a>
|
|
cannam@95
|
60 In certain cases, it may be advantageous to combine MPI
|
|
cannam@95
|
61 (distributed-memory) and threads (shared-memory) parallelization.
|
|
cannam@95
|
62 FFTW supports this, with certain caveats. For example, if you have a
|
|
cannam@95
|
63 cluster of 4-processor shared-memory nodes, you may want to use
|
|
cannam@95
|
64 threads within the nodes and MPI between the nodes, instead of MPI for
|
|
cannam@95
|
65 all parallelization.
|
|
cannam@95
|
66
|
|
cannam@95
|
67 <p>In particular, it is possible to seamlessly combine the MPI FFTW
|
|
cannam@95
|
68 routines with the multi-threaded FFTW routines (see <a href="Multi_002dthreaded-FFTW.html#Multi_002dthreaded-FFTW">Multi-threaded FFTW</a>). However, some care must be taken in the initialization code,
|
|
cannam@95
|
69 which should look something like this:
|
|
cannam@95
|
70
|
|
cannam@95
|
71 <pre class="example"> int threads_ok;
|
|
cannam@95
|
72
|
|
cannam@95
|
73 int main(int argc, char **argv)
|
|
cannam@95
|
74 {
|
|
cannam@95
|
75 int provided;
|
|
cannam@95
|
76 MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
|
|
cannam@95
|
77 threads_ok = provided >= MPI_THREAD_FUNNELED;
|
|
cannam@95
|
78
|
|
cannam@95
|
79 if (threads_ok) threads_ok = fftw_init_threads();
|
|
cannam@95
|
80 fftw_mpi_init();
|
|
cannam@95
|
81
|
|
cannam@95
|
82 ...
|
|
cannam@95
|
83 if (threads_ok) fftw_plan_with_nthreads(...);
|
|
cannam@95
|
84 ...
|
|
cannam@95
|
85
|
|
cannam@95
|
86 MPI_Finalize();
|
|
cannam@95
|
87 }
|
|
cannam@95
|
88 </pre>
|
|
cannam@95
|
89 <p><a name="index-fftw_005fmpi_005finit-428"></a><a name="index-fftw_005finit_005fthreads-429"></a><a name="index-fftw_005fplan_005fwith_005fnthreads-430"></a>
|
|
cannam@95
|
90 First, note that instead of calling <code>MPI_Init</code>, you should call
|
|
cannam@95
|
91 <code>MPI_Init_threads</code>, which is the initialization routine defined
|
|
cannam@95
|
92 by the MPI-2 standard to indicate to MPI that your program will be
|
|
cannam@95
|
93 multithreaded. We pass <code>MPI_THREAD_FUNNELED</code>, which indicates
|
|
cannam@95
|
94 that we will only call MPI routines from the main thread. (FFTW will
|
|
cannam@95
|
95 launch additional threads internally, but the extra threads will not
|
|
cannam@95
|
96 call MPI code.) (You may also pass <code>MPI_THREAD_SERIALIZED</code> or
|
|
cannam@95
|
97 <code>MPI_THREAD_MULTIPLE</code>, which requests additional multithreading
|
|
cannam@95
|
98 support from the MPI implementation, but this is not required by
|
|
cannam@95
|
99 FFTW.) The <code>provided</code> parameter returns what level of threads
|
|
cannam@95
|
100 support is actually supported by your MPI implementation; this
|
|
cannam@95
|
101 <em>must</em> be at least <code>MPI_THREAD_FUNNELED</code> if you want to call
|
|
cannam@95
|
102 the FFTW threads routines, so we define a global variable
|
|
cannam@95
|
103 <code>threads_ok</code> to record this. You should only call
|
|
cannam@95
|
104 <code>fftw_init_threads</code> or <code>fftw_plan_with_nthreads</code> if
|
|
cannam@95
|
105 <code>threads_ok</code> is true. For more information on thread safety in
|
|
cannam@95
|
106 MPI, see the
|
|
cannam@95
|
107 <a href="http://www.mpi-forum.org/docs/mpi-20-html/node162.htm">MPI and Threads</a> section of the MPI-2 standard.
|
|
cannam@95
|
108 <a name="index-thread-safety-431"></a>
|
|
cannam@95
|
109
|
|
cannam@95
|
110 <p>Second, we must call <code>fftw_init_threads</code> <em>before</em>
|
|
cannam@95
|
111 <code>fftw_mpi_init</code>. This is critical for technical reasons having
|
|
cannam@95
|
112 to do with how FFTW initializes its list of algorithms.
|
|
cannam@95
|
113
|
|
cannam@95
|
114 <p>Then, if you call <code>fftw_plan_with_nthreads(N)</code>, <em>every</em> MPI
|
|
cannam@95
|
115 process will launch (up to) <code>N</code> threads to parallelize its transforms.
|
|
cannam@95
|
116
|
|
cannam@95
|
117 <p>For example, in the hypothetical cluster of 4-processor nodes, you
|
|
cannam@95
|
118 might wish to launch only a single MPI process per node, and then call
|
|
cannam@95
|
119 <code>fftw_plan_with_nthreads(4)</code> on each process to use all
|
|
cannam@95
|
120 processors in the nodes.
|
|
cannam@95
|
121
|
|
cannam@95
|
122 <p>This may or may not be faster than simply using as many MPI processes
|
|
cannam@95
|
123 as you have processors, however. On the one hand, using threads
|
|
cannam@95
|
124 within a node eliminates the need for explicit message passing within
|
|
cannam@95
|
125 the node. On the other hand, FFTW's transpose routines are not
|
|
cannam@95
|
126 multi-threaded, and this means that the communications that do take
|
|
cannam@95
|
127 place will not benefit from parallelization within the node.
|
|
cannam@95
|
128 Moreover, many MPI implementations already have optimizations to
|
|
cannam@95
|
129 exploit shared memory when it is available, so adding the
|
|
cannam@95
|
130 multithreaded FFTW on top of this may be superfluous.
|
|
cannam@95
|
131 <a name="index-transpose-432"></a>
|
|
cannam@95
|
132 <!-- -->
|
|
cannam@95
|
133
|
|
cannam@95
|
134 </body></html>
|
|
cannam@95
|
135
|
