yading@10: \input texinfo @c -*- texinfo -*- yading@10: yading@10: @settitle FFmpeg Automated Testing Environment yading@10: @titlepage yading@10: @center @titlefont{FFmpeg Automated Testing Environment} yading@10: @end titlepage yading@10: yading@10: @node Top yading@10: @top yading@10: yading@10: @contents yading@10: yading@10: @chapter Introduction yading@10: yading@10: FATE is an extended regression suite on the client-side and a means yading@10: for results aggregation and presentation on the server-side. yading@10: yading@10: The first part of this document explains how you can use FATE from yading@10: your FFmpeg source directory to test your ffmpeg binary. The second yading@10: part describes how you can run FATE to submit the results to FFmpeg's yading@10: FATE server. yading@10: yading@10: In any way you can have a look at the publicly viewable FATE results yading@10: by visiting this website: yading@10: yading@10: @url{http://fate.ffmpeg.org/} yading@10: yading@10: This is especially recommended for all people contributing source yading@10: code to FFmpeg, as it can be seen if some test on some platform broke yading@10: with their recent contribution. This usually happens on the platforms yading@10: the developers could not test on. yading@10: yading@10: The second part of this document describes how you can run FATE to yading@10: submit your results to FFmpeg's FATE server. If you want to submit your yading@10: results be sure to check that your combination of CPU, OS and compiler yading@10: is not already listed on the above mentioned website. yading@10: yading@10: In the third part you can find a comprehensive listing of FATE makefile yading@10: targets and variables. yading@10: yading@10: yading@10: @chapter Using FATE from your FFmpeg source directory yading@10: yading@10: If you want to run FATE on your machine you need to have the samples yading@10: in place. You can get the samples via the build target fate-rsync. yading@10: Use this command from the top-level source directory: yading@10: yading@10: @example yading@10: make fate-rsync SAMPLES=fate-suite/ yading@10: make fate SAMPLES=fate-suite/ yading@10: @end example yading@10: yading@10: The above commands set the samples location by passing a makefile yading@10: variable via command line. It is also possible to set the samples yading@10: location at source configuration time by invoking configure with yading@10: `--samples='. Afterwards you can yading@10: invoke the makefile targets without setting the SAMPLES makefile yading@10: variable. This is illustrated by the following commands: yading@10: yading@10: @example yading@10: ./configure --samples=fate-suite/ yading@10: make fate-rsync yading@10: make fate yading@10: @end example yading@10: yading@10: Yet another way to tell FATE about the location of the sample yading@10: directory is by making sure the environment variable FATE_SAMPLES yading@10: contains the path to your samples directory. This can be achieved yading@10: by e.g. putting that variable in your shell profile or by setting yading@10: it in your interactive session. yading@10: yading@10: @example yading@10: FATE_SAMPLES=fate-suite/ make fate yading@10: @end example yading@10: yading@10: @float NOTE yading@10: Do not put a '~' character in the samples path to indicate a home yading@10: directory. Because of shell nuances, this will cause FATE to fail. yading@10: @end float yading@10: yading@10: To use a custom wrapper to run the test, pass @option{--target-exec} to yading@10: @command{configure} or set the @var{TARGET_EXEC} Make variable. yading@10: yading@10: yading@10: @chapter Submitting the results to the FFmpeg result aggregation server yading@10: yading@10: To submit your results to the server you should run fate through the yading@10: shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs yading@10: to be invoked with a configuration file as its first argument. yading@10: yading@10: @example yading@10: tests/fate.sh /path/to/fate_config yading@10: @end example yading@10: yading@10: A configuration file template with comments describing the individual yading@10: configuration variables can be found at @file{doc/fate_config.sh.template}. yading@10: yading@10: @ifhtml yading@10: The mentioned configuration template is also available here: yading@10: @verbatiminclude fate_config.sh.template yading@10: @end ifhtml yading@10: yading@10: Create a configuration that suits your needs, based on the configuration yading@10: template. The `slot' configuration variable can be any string that is not yading@10: yet used, but it is suggested that you name it adhering to the following yading@10: pattern ---. The configuration file yading@10: itself will be sourced in a shell script, therefore all shell features may yading@10: be used. This enables you to setup the environment as you need it for your yading@10: build. yading@10: yading@10: For your first test runs the `fate_recv' variable should be empty or yading@10: commented out. This will run everything as normal except that it will omit yading@10: the submission of the results to the server. The following files should be yading@10: present in $workdir as specified in the configuration file: yading@10: yading@10: @itemize yading@10: @item configure.log yading@10: @item compile.log yading@10: @item test.log yading@10: @item report yading@10: @item version yading@10: @end itemize yading@10: yading@10: When you have everything working properly you can create an SSH key pair yading@10: and send the public key to the FATE server administrator who can be contacted yading@10: at the email address @email{fate-admin@@ffmpeg.org}. yading@10: yading@10: Configure your SSH client to use public key authentication with that key yading@10: when connecting to the FATE server. Also do not forget to check the identity yading@10: of the server and to accept its host key. This can usually be achieved by yading@10: running your SSH client manually and killing it after you accepted the key. yading@10: The FATE server's fingerprint is: yading@10: yading@10: b1:31:c8:79:3f:04:1d:f8:f2:23:26:5a:fd:55:fa:92 yading@10: yading@10: If you have problems connecting to the FATE server, it may help to try out yading@10: the @command{ssh} command with one or more @option{-v} options. You should yading@10: get detailed output concerning your SSH configuration and the authentication yading@10: process. yading@10: yading@10: The only thing left is to automate the execution of the fate.sh script and yading@10: the synchronisation of the samples directory. yading@10: yading@10: yading@10: @chapter FATE makefile targets and variables yading@10: yading@10: @section Makefile targets yading@10: yading@10: @table @option yading@10: @item fate-rsync yading@10: Download/synchronize sample files to the configured samples directory. yading@10: yading@10: @item fate-list yading@10: Will list all fate/regression test targets. yading@10: yading@10: @item fate yading@10: Run the FATE test suite (requires the fate-suite dataset). yading@10: @end table yading@10: yading@10: @section Makefile variables yading@10: yading@10: @table @option yading@10: @item V yading@10: Verbosity level, can be set to 0, 1 or 2. yading@10: @itemize yading@10: @item 0: show just the test arguments yading@10: @item 1: show just the command used in the test yading@10: @item 2: show everything yading@10: @end itemize yading@10: yading@10: @item SAMPLES yading@10: Specify or override the path to the FATE samples at make time, it has a yading@10: meaning only while running the regression tests. yading@10: yading@10: @item THREADS yading@10: Specify how many threads to use while running regression tests, it is yading@10: quite useful to detect thread-related regressions. yading@10: @item THREAD_TYPE yading@10: Specify which threading strategy test, either @var{slice} or @var{frame}, yading@10: by default @var{slice+frame} yading@10: @item CPUFLAGS yading@10: Specify CPU flags. yading@10: @item TARGET_EXEC yading@10: Specify or override the wrapper used to run the tests. yading@10: The @var{TARGET_EXEC} option provides a way to run FATE wrapped in yading@10: @command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets yading@10: through @command{ssh}. yading@10: @item GEN yading@10: Set to @var{1} to generate the missing or mismatched references. yading@10: @end table yading@10: yading@10: @section Examples yading@10: yading@10: @example yading@10: make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate yading@10: @end example