cannam@95: @node Multi-threaded FFTW, Distributed-memory FFTW with MPI, FFTW Reference, Top cannam@95: @chapter Multi-threaded FFTW cannam@95: cannam@95: @cindex parallel transform cannam@95: In this chapter we document the parallel FFTW routines for cannam@95: shared-memory parallel hardware. These routines, which support cannam@95: parallel one- and multi-dimensional transforms of both real and cannam@95: complex data, are the easiest way to take advantage of multiple cannam@95: processors with FFTW. They work just like the corresponding cannam@95: uniprocessor transform routines, except that you have an extra cannam@95: initialization routine to call, and there is a routine to set the cannam@95: number of threads to employ. Any program that uses the uniprocessor cannam@95: FFTW can therefore be trivially modified to use the multi-threaded cannam@95: FFTW. cannam@95: cannam@95: A shared-memory machine is one in which all CPUs can directly access cannam@95: the same main memory, and such machines are now common due to the cannam@95: ubiquity of multi-core CPUs. FFTW's multi-threading support allows cannam@95: you to utilize these additional CPUs transparently from a single cannam@95: program. However, this does not necessarily translate into cannam@95: performance gains---when multiple threads/CPUs are employed, there is cannam@95: an overhead required for synchronization that may outweigh the cannam@95: computatational parallelism. Therefore, you can only benefit from cannam@95: threads if your problem is sufficiently large. cannam@95: @cindex shared-memory cannam@95: @cindex threads cannam@95: cannam@95: @menu cannam@95: * Installation and Supported Hardware/Software:: cannam@95: * Usage of Multi-threaded FFTW:: cannam@95: * How Many Threads to Use?:: cannam@95: * Thread safety:: cannam@95: @end menu cannam@95: cannam@95: @c ------------------------------------------------------------ cannam@95: @node Installation and Supported Hardware/Software, Usage of Multi-threaded FFTW, Multi-threaded FFTW, Multi-threaded FFTW cannam@95: @section Installation and Supported Hardware/Software cannam@95: cannam@95: All of the FFTW threads code is located in the @code{threads} cannam@95: subdirectory of the FFTW package. On Unix systems, the FFTW threads cannam@95: libraries and header files can be automatically configured, compiled, cannam@95: and installed along with the uniprocessor FFTW libraries simply by cannam@95: including @code{--enable-threads} in the flags to the @code{configure} cannam@95: script (@pxref{Installation on Unix}), or @code{--enable-openmp} to use cannam@95: @uref{http://www.openmp.org,OpenMP} threads. cannam@95: @fpindex configure cannam@95: cannam@95: cannam@95: @cindex portability cannam@95: @cindex OpenMP cannam@95: The threads routines require your operating system to have some sort cannam@95: of shared-memory threads support. Specifically, the FFTW threads cannam@95: package works with POSIX threads (available on most Unix variants, cannam@95: from GNU/Linux to MacOS X) and Win32 threads. OpenMP threads, which cannam@95: are supported in many common compilers (e.g. gcc) are also supported, cannam@95: and may give better performance on some systems. (OpenMP threads are cannam@95: also useful if you are employing OpenMP in your own code, in order to cannam@95: minimize conflicts between threading models.) If you have a cannam@95: shared-memory machine that uses a different threads API, it should be cannam@95: a simple matter of programming to include support for it; see the file cannam@95: @code{threads/threads.c} for more detail. cannam@95: cannam@95: You can compile FFTW with @emph{both} @code{--enable-threads} and cannam@95: @code{--enable-openmp} at the same time, since they install libraries cannam@95: with different names (@samp{fftw3_threads} and @samp{fftw3_omp}, as cannam@95: described below). However, your programs may only link to @emph{one} cannam@95: of these two libraries at a time. cannam@95: cannam@95: Ideally, of course, you should also have multiple processors in order to cannam@95: get any benefit from the threaded transforms. cannam@95: cannam@95: @c ------------------------------------------------------------ cannam@95: @node Usage of Multi-threaded FFTW, How Many Threads to Use?, Installation and Supported Hardware/Software, Multi-threaded FFTW cannam@95: @section Usage of Multi-threaded FFTW cannam@95: cannam@95: Here, it is assumed that the reader is already familiar with the usage cannam@95: of the uniprocessor FFTW routines, described elsewhere in this manual. cannam@95: We only describe what one has to change in order to use the cannam@95: multi-threaded routines. cannam@95: cannam@95: @cindex OpenMP cannam@95: First, programs using the parallel complex transforms should be linked cannam@95: with @code{-lfftw3_threads -lfftw3 -lm} on Unix, or @code{-lfftw3_omp cannam@95: -lfftw3 -lm} if you compiled with OpenMP. You will also need to link cannam@95: with whatever library is responsible for threads on your system cannam@95: (e.g. @code{-lpthread} on GNU/Linux) or include whatever compiler flag cannam@95: enables OpenMP (e.g. @code{-fopenmp} with gcc). cannam@95: @cindex linking on Unix cannam@95: cannam@95: cannam@95: Second, before calling @emph{any} FFTW routines, you should call the cannam@95: function: cannam@95: cannam@95: @example cannam@95: int fftw_init_threads(void); cannam@95: @end example cannam@95: @findex fftw_init_threads cannam@95: cannam@95: This function, which need only be called once, performs any one-time cannam@95: initialization required to use threads on your system. It returns zero cannam@95: if there was some error (which should not happen under normal cannam@95: circumstances) and a non-zero value otherwise. cannam@95: cannam@95: Third, before creating a plan that you want to parallelize, you should cannam@95: call: cannam@95: cannam@95: @example cannam@95: void fftw_plan_with_nthreads(int nthreads); cannam@95: @end example cannam@95: @findex fftw_plan_with_nthreads cannam@95: cannam@95: The @code{nthreads} argument indicates the number of threads you want cannam@95: FFTW to use (or actually, the maximum number). All plans subsequently cannam@95: created with any planner routine will use that many threads. You can cannam@95: call @code{fftw_plan_with_nthreads}, create some plans, call cannam@95: @code{fftw_plan_with_nthreads} again with a different argument, and cannam@95: create some more plans for a new number of threads. Plans already created cannam@95: before a call to @code{fftw_plan_with_nthreads} are unaffected. If you cannam@95: pass an @code{nthreads} argument of @code{1} (the default), threads are cannam@95: disabled for subsequent plans. cannam@95: cannam@95: @cindex OpenMP cannam@95: With OpenMP, to configure FFTW to use all of the currently running cannam@95: OpenMP threads (set by @code{omp_set_num_threads(nthreads)} or by the cannam@95: @code{OMP_NUM_THREADS} environment variable), you can do: cannam@95: @code{fftw_plan_with_nthreads(omp_get_max_threads())}. (The @samp{omp_} cannam@95: OpenMP functions are declared via @code{#include }.) cannam@95: cannam@95: @cindex thread safety cannam@95: Given a plan, you then execute it as usual with cannam@95: @code{fftw_execute(plan)}, and the execution will use the number of cannam@95: threads specified when the plan was created. When done, you destroy cannam@95: it as usual with @code{fftw_destroy_plan}. As described in cannam@95: @ref{Thread safety}, plan @emph{execution} is thread-safe, but plan cannam@95: creation and destruction are @emph{not}: you should create/destroy cannam@95: plans only from a single thread, but can safely execute multiple plans cannam@95: in parallel. cannam@95: cannam@95: There is one additional routine: if you want to get rid of all memory cannam@95: and other resources allocated internally by FFTW, you can call: cannam@95: cannam@95: @example cannam@95: void fftw_cleanup_threads(void); cannam@95: @end example cannam@95: @findex fftw_cleanup_threads cannam@95: cannam@95: which is much like the @code{fftw_cleanup()} function except that it cannam@95: also gets rid of threads-related data. You must @emph{not} execute any cannam@95: previously created plans after calling this function. cannam@95: cannam@95: We should also mention one other restriction: if you save wisdom from a cannam@95: program using the multi-threaded FFTW, that wisdom @emph{cannot be used} cannam@95: by a program using only the single-threaded FFTW (i.e. not calling cannam@95: @code{fftw_init_threads}). @xref{Words of Wisdom-Saving Plans}. cannam@95: cannam@95: @c ------------------------------------------------------------ cannam@95: @node How Many Threads to Use?, Thread safety, Usage of Multi-threaded FFTW, Multi-threaded FFTW cannam@95: @section How Many Threads to Use? cannam@95: cannam@95: @cindex number of threads cannam@95: There is a fair amount of overhead involved in synchronizing threads, cannam@95: so the optimal number of threads to use depends upon the size of the cannam@95: transform as well as on the number of processors you have. cannam@95: cannam@95: As a general rule, you don't want to use more threads than you have cannam@95: processors. (Using more threads will work, but there will be extra cannam@95: overhead with no benefit.) In fact, if the problem size is too small, cannam@95: you may want to use fewer threads than you have processors. cannam@95: cannam@95: You will have to experiment with your system to see what level of cannam@95: parallelization is best for your problem size. Typically, the problem cannam@95: will have to involve at least a few thousand data points before threads cannam@95: become beneficial. If you plan with @code{FFTW_PATIENT}, it will cannam@95: automatically disable threads for sizes that don't benefit from cannam@95: parallelization. cannam@95: @ctindex FFTW_PATIENT cannam@95: cannam@95: @c ------------------------------------------------------------ cannam@95: @node Thread safety, , How Many Threads to Use?, Multi-threaded FFTW cannam@95: @section Thread safety cannam@95: cannam@95: @cindex threads cannam@95: @cindex OpenMP cannam@95: @cindex thread safety cannam@95: Users writing multi-threaded programs (including OpenMP) must concern cannam@95: themselves with the @dfn{thread safety} of the libraries they cannam@95: use---that is, whether it is safe to call routines in parallel from cannam@95: multiple threads. FFTW can be used in such an environment, but some cannam@95: care must be taken because the planner routines share data cannam@95: (e.g. wisdom and trigonometric tables) between calls and plans. cannam@95: cannam@95: The upshot is that the only thread-safe (re-entrant) routine in FFTW is cannam@95: @code{fftw_execute} (and the new-array variants thereof). All other routines cannam@95: (e.g. the planner) should only be called from one thread at a time. So, cannam@95: for example, you can wrap a semaphore lock around any calls to the cannam@95: planner; even more simply, you can just create all of your plans from cannam@95: one thread. We do not think this should be an important restriction cannam@95: (FFTW is designed for the situation where the only performance-sensitive cannam@95: code is the actual execution of the transform), and the benefits of cannam@95: shared data between plans are great. cannam@95: cannam@95: Note also that, since the plan is not modified by @code{fftw_execute}, cannam@95: it is safe to execute the @emph{same plan} in parallel by multiple cannam@95: threads. However, since a given plan operates by default on a fixed cannam@95: array, you need to use one of the new-array execute functions (@pxref{New-array Execute Functions}) so that different threads compute the transform of different data. cannam@95: cannam@95: (Users should note that these comments only apply to programs using cannam@95: shared-memory threads or OpenMP. Parallelism using MPI or forked processes cannam@95: involves a separate address-space and global variables for each process, cannam@95: and is not susceptible to problems of this sort.) cannam@95: cannam@95: If you are configured FFTW with the @code{--enable-debug} or cannam@95: @code{--enable-debug-malloc} flags (@pxref{Installation on Unix}), cannam@95: then @code{fftw_execute} is not thread-safe. These flags are not cannam@95: documented because they are intended only for developing cannam@95: and debugging FFTW, but if you must use @code{--enable-debug} then you cannam@95: should also specifically pass @code{--disable-debug-malloc} for cannam@95: @code{fftw_execute} to be thread-safe. cannam@95: