yading@10: FFmpeg Automated Testing Environment yading@10: ************************************ yading@10: yading@10: Table of Contents yading@10: ***************** yading@10: yading@10: FFmpeg Automated Testing Environment yading@10: 1 Introduction yading@10: 2 Using FATE from your FFmpeg source directory yading@10: 3 Submitting the results to the FFmpeg result aggregation server yading@10: 4 FATE makefile targets and variables yading@10: 4.1 Makefile targets yading@10: 4.2 Makefile variables yading@10: 4.3 Examples yading@10: yading@10: yading@10: 1 Introduction yading@10: ************** yading@10: yading@10: FATE is an extended regression suite on the client-side and a means for yading@10: results aggregation and presentation on the server-side. yading@10: yading@10: The first part of this document explains how you can use FATE from yading@10: your FFmpeg source directory to test your ffmpeg binary. The second yading@10: part describes how you can run FATE to submit the results to FFmpeg's yading@10: FATE server. yading@10: yading@10: In any way you can have a look at the publicly viewable FATE results yading@10: by visiting this website: yading@10: yading@10: `http://fate.ffmpeg.org/' yading@10: yading@10: This is especially recommended for all people contributing source yading@10: code to FFmpeg, as it can be seen if some test on some platform broke yading@10: with their recent contribution. This usually happens on the platforms yading@10: the developers could not test on. yading@10: yading@10: The second part of this document describes how you can run FATE to yading@10: submit your results to FFmpeg's FATE server. If you want to submit your yading@10: results be sure to check that your combination of CPU, OS and compiler yading@10: is not already listed on the above mentioned website. yading@10: yading@10: In the third part you can find a comprehensive listing of FATE yading@10: makefile targets and variables. yading@10: yading@10: 2 Using FATE from your FFmpeg source directory yading@10: ********************************************** yading@10: yading@10: If you want to run FATE on your machine you need to have the samples in yading@10: place. You can get the samples via the build target fate-rsync. Use yading@10: this command from the top-level source directory: yading@10: yading@10: make fate-rsync SAMPLES=fate-suite/ yading@10: make fate SAMPLES=fate-suite/ yading@10: yading@10: The above commands set the samples location by passing a makefile yading@10: variable via command line. It is also possible to set the samples yading@10: location at source configuration time by invoking configure with yading@10: `-samples='. Afterwards you can invoke yading@10: the makefile targets without setting the SAMPLES makefile variable. yading@10: This is illustrated by the following commands: yading@10: yading@10: ./configure --samples=fate-suite/ yading@10: make fate-rsync yading@10: make fate yading@10: yading@10: Yet another way to tell FATE about the location of the sample yading@10: directory is by making sure the environment variable FATE_SAMPLES yading@10: contains the path to your samples directory. This can be achieved by yading@10: e.g. putting that variable in your shell profile or by setting it in yading@10: your interactive session. yading@10: yading@10: FATE_SAMPLES=fate-suite/ make fate yading@10: yading@10: Do not put a '~' character in the samples path to indicate a home yading@10: directory. Because of shell nuances, this will cause FATE to fail. yading@10: yading@10: NOTE yading@10: yading@10: To use a custom wrapper to run the test, pass `--target-exec' to yading@10: `configure' or set the TARGET_EXEC Make variable. yading@10: yading@10: 3 Submitting the results to the FFmpeg result aggregation server yading@10: **************************************************************** yading@10: yading@10: To submit your results to the server you should run fate through the yading@10: shell script `tests/fate.sh' from the FFmpeg sources. This script needs yading@10: to be invoked with a configuration file as its first argument. yading@10: yading@10: tests/fate.sh /path/to/fate_config yading@10: yading@10: A configuration file template with comments describing the individual yading@10: configuration variables can be found at `doc/fate_config.sh.template'. yading@10: yading@10: Create a configuration that suits your needs, based on the yading@10: configuration template. The `slot' configuration variable can be any yading@10: string that is not yet used, but it is suggested that you name it yading@10: adhering to the following pattern ---. The configuration file itself will be sourced in a shell yading@10: script, therefore all shell features may be used. This enables you to yading@10: setup the environment as you need it for your build. yading@10: yading@10: For your first test runs the `fate_recv' variable should be empty or yading@10: commented out. This will run everything as normal except that it will yading@10: omit the submission of the results to the server. The following files yading@10: should be present in $workdir as specified in the configuration file: yading@10: yading@10: * configure.log yading@10: yading@10: * compile.log yading@10: yading@10: * test.log yading@10: yading@10: * report yading@10: yading@10: * version yading@10: yading@10: When you have everything working properly you can create an SSH key yading@10: pair and send the public key to the FATE server administrator who can yading@10: be contacted at the email address . yading@10: yading@10: Configure your SSH client to use public key authentication with that yading@10: key when connecting to the FATE server. Also do not forget to check the yading@10: identity of the server and to accept its host key. This can usually be yading@10: achieved by running your SSH client manually and killing it after you yading@10: accepted the key. The FATE server's fingerprint is: yading@10: yading@10: b1:31:c8:79:3f:04:1d:f8:f2:23:26:5a:fd:55:fa:92 yading@10: yading@10: If you have problems connecting to the FATE server, it may help to yading@10: try out the `ssh' command with one or more `-v' options. You should get yading@10: detailed output concerning your SSH configuration and the authentication yading@10: process. yading@10: yading@10: The only thing left is to automate the execution of the fate.sh yading@10: script and the synchronisation of the samples directory. yading@10: yading@10: 4 FATE makefile targets and variables yading@10: ************************************* yading@10: yading@10: 4.1 Makefile targets yading@10: ==================== yading@10: yading@10: `fate-rsync' yading@10: Download/synchronize sample files to the configured samples yading@10: directory. yading@10: yading@10: `fate-list' yading@10: Will list all fate/regression test targets. yading@10: yading@10: `fate' yading@10: Run the FATE test suite (requires the fate-suite dataset). yading@10: yading@10: 4.2 Makefile variables yading@10: ====================== yading@10: yading@10: `V' yading@10: Verbosity level, can be set to 0, 1 or 2. yading@10: * 0: show just the test arguments yading@10: yading@10: * 1: show just the command used in the test yading@10: yading@10: * 2: show everything yading@10: yading@10: `SAMPLES' yading@10: Specify or override the path to the FATE samples at make time, it yading@10: has a meaning only while running the regression tests. yading@10: yading@10: `THREADS' yading@10: Specify how many threads to use while running regression tests, it yading@10: is quite useful to detect thread-related regressions. yading@10: yading@10: `THREAD_TYPE' yading@10: Specify which threading strategy test, either SLICE or FRAME, yading@10: by default SLICE+FRAME yading@10: yading@10: `CPUFLAGS' yading@10: Specify CPU flags. yading@10: yading@10: `TARGET_EXEC' yading@10: Specify or override the wrapper used to run the tests. The yading@10: TARGET_EXEC option provides a way to run FATE wrapped in yading@10: `valgrind', `qemu-user' or `wine' or on remote targets through yading@10: `ssh'. yading@10: yading@10: `GEN' yading@10: Set to 1 to generate the missing or mismatched references. yading@10: yading@10: 4.3 Examples yading@10: ============ yading@10: yading@10: make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate yading@10: