README.md 19.8 KB
Newer Older
Tom Finegan's avatar
Tom Finegan committed
1 2
# AV1 Codec Library

3 4 5
## Contents
1. [Building the lib and applications](#building-the-library-and-applications)
    - [Prerequisites](#prerequisites)
6
    - [Get the code](#get-the-code)
7 8 9 10 11 12 13 14 15
    - [Basics](#basic-build)
    - [Configuration options](#configuration-options)
    - [Dylib builds](#dylib-builds)
    - [Debugging](#debugging)
    - [Cross compiling](#cross-compiling)
    - [Sanitizer support](#sanitizers)
    - [MSVC builds](#microsoft-visual-studio-builds)
    - [Xcode builds](#xcode-builds)
    - [Emscripten builds](#emscripten-builds)
16
    - [Extra Build Flags](#extra-build-flags)
17 18 19 20
2. [Testing the library](#testing-the-av1-codec)
    - [Basics](#testing-basics)
        - [Unit tests](#1_unit-tests)
        - [Example tests](#2_example-tests)
21
        - [Encoder tests](#3_encoder-tests)
22 23
    - [IDE hosted tests](#ide-hosted-tests)
    - [Downloading test data](#downloading-the-test-data)
24
    - [Additional test data](#additional-test-data)
25 26 27 28
    - [Sharded testing](#sharded-testing)
        - [Running tests directly](#1_running-test_libaom-directly)
        - [Running tests via CMake](#2_running-the-tests-via-the-cmake-build)
3. [Coding style](#coding-style)
29 30 31 32 33 34 35 36 37 38 39
4. [Submitting patches](#submitting-patches)
    - [Login cookie](#login-cookie)
    - [Contributor agreement](#contributor-agreement)
    - [Testing your code](#testing-your-code)
    - [Commit message hook](#commit-message-hook)
    - [Upload your change](#upload-your-change)
    - [Incorporating Reviewer Comments](#incorporating-reviewer-comments)
    - [Submitting your change](#submitting-your-change)
    - [Viewing change status](#viewing-the-status-of-uploaded-changes)
5. [Support](#support)
6. [Bug reports](#bug-reports)
40

Tom Finegan's avatar
Tom Finegan committed
41 42 43 44 45 46 47 48 49 50
## Building the library and applications

### Prerequisites

 1. [CMake](https://cmake.org) version 3.5 or higher.
 2. [Git](https://git-scm.com/).
 3. [Perl](https://www.perl.org/).
 4. For x86 targets, [yasm](http://yasm.tortall.net/), which is preferred, or a
    recent version of [nasm](http://www.nasm.us/).
 5. Building the documentation requires [doxygen](http://doxygen.org).
51
 6. Building the unit tests requires [Python](https://www.python.org/).
52 53
 7. Emscripten builds require the portable
   [EMSDK](https://kripken.github.io/emscripten-site/index.html).
Tom Finegan's avatar
Tom Finegan committed
54

55 56 57 58 59 60 61 62 63 64 65
### Get the code

The AV1 library source code is stored in the Alliance for Open Media Git
repository:

~~~
    $ git clone https://aomedia.googlesource.com/aom
    # By default, the above command stores the source in the aom directory:
    $ cd aom
~~~

Tom Finegan's avatar
Tom Finegan committed
66 67 68 69 70 71 72
### Basic build

CMake replaces the configure step typical of many projects. Running CMake will
produce configuration and build files for the currently selected CMake
generator. For most systems the default generator is Unix Makefiles. The basic
form of a makefile build is the following:

73
~~~
Tom Finegan's avatar
Tom Finegan committed
74 75
    $ cmake path/to/aom
    $ make
76
~~~
Tom Finegan's avatar
Tom Finegan committed
77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92

The above will generate a makefile build that produces the AV1 library and
applications for the current host system after the make step completes
successfully. The compiler chosen varies by host platform, but a general rule
applies: On systems where cc and c++ are present in $PATH at the time CMake is
run the generated build will use cc and c++ by default.

### Configuration options

The AV1 codec library has a great many configuration options. These come in two
varieties:

 1. Build system configuration options. These have the form `ENABLE_FEATURE`.
 2. AV1 codec configuration options. These have the form `CONFIG_FEATURE`.

Both types of options are set at the time CMake is run. The following example
93
enables ccache and disables the AV1 encoder:
Tom Finegan's avatar
Tom Finegan committed
94

95
~~~
96
    $ cmake path/to/aom -DENABLE_CCACHE=1 -DCONFIG_AV1_ENCODER=0
Tom Finegan's avatar
Tom Finegan committed
97
    $ make
98
~~~
Tom Finegan's avatar
Tom Finegan committed
99 100 101 102 103 104

The available configuration options are too numerous to list here. Build system
configuration options can be found at the top of the CMakeLists.txt file found
in the root of the AV1 repository, and AV1 codec configuration options can
currently be found in the file `build/cmake/aom_config_defaults.cmake`.

105 106 107
### Dylib builds

A dylib (shared object) build of the AV1 codec library can be enabled via the
108
CMake built in variable `BUILD_SHARED_LIBS`:
109

110 111
~~~
    $ cmake path/to/aom -DBUILD_SHARED_LIBS=1
112
    $ make
113
~~~
114 115 116

This is currently only supported on non-Windows targets.

117 118 119 120 121 122 123 124 125 126 127
### Debugging

Depending on the generator used there are multiple ways of going about
debugging AV1 components. For single configuration generators like the Unix
Makefiles generator, setting `CMAKE_BUILD_TYPE` to Debug is sufficient:

~~~
    $ cmake path/to/aom -DCMAKE_BUILD_TYPE=Debug
~~~

For Xcode, mainly because configuration controls for Xcode builds are buried two
Tom Finegan's avatar
Tom Finegan committed
128 129
configuration windows deep and must be set for each subproject within the Xcode
IDE individually, `CMAKE_CONFIGURATION_TYPES` should be set to Debug:
130 131 132 133 134 135 136 137 138

~~~
    $ cmake path/to/aom -G Xcode -DCMAKE_CONFIGURATION_TYPES=Debug
~~~

For Visual Studio the in-IDE configuration controls should be used. Simply set
the IDE project configuration to Debug to allow for stepping through the code.

In addition to the above it can sometimes be useful to debug only C and C++
Tom Finegan's avatar
Tom Finegan committed
139
code. To disable all assembly code and intrinsics set `AOM_TARGET_CPU` to
140 141 142 143 144 145
generic at generation time:

~~~
    $ cmake path/to/aom -DAOM_TARGET_CPU=generic
~~~

Tom Finegan's avatar
Tom Finegan committed
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163
### Cross compiling

For the purposes of building the AV1 codec and applications and relative to the
scope of this guide, all builds for architectures differing from the native host
architecture will be considered cross compiles. The AV1 CMake build handles
cross compiling via the use of toolchain files included in the AV1 repository.
The toolchain files available at the time of this writing are:

 - arm64-ios.cmake
 - arm64-linux-gcc.cmake
 - armv7-ios.cmake
 - armv7-linux-gcc.cmake
 - armv7s-ios.cmake
 - mips32-linux-gcc.cmake
 - mips64-linux-gcc.cmake
 - x86-ios-simulator.cmake
 - x86-linux.cmake
 - x86-macos.cmake
Tom Finegan's avatar
Tom Finegan committed
164
 - x86-mingw-gcc.cmake
Tom Finegan's avatar
Tom Finegan committed
165
 - x86\_64-ios-simulator.cmake
Tom Finegan's avatar
Tom Finegan committed
166
 - x86\_64-mingw-gcc.cmake
Tom Finegan's avatar
Tom Finegan committed
167 168 169 170

The following example demonstrates use of the x86-macos.cmake toolchain file on
a x86\_64 MacOS host:

171
~~~
Tom Finegan's avatar
Tom Finegan committed
172 173 174
    $ cmake path/to/aom \
      -DCMAKE_TOOLCHAIN_FILE=path/to/aom/build/cmake/toolchains/x86-macos.cmake
    $ make
175
~~~
Tom Finegan's avatar
Tom Finegan committed
176 177 178 179 180 181 182 183 184

To build for an unlisted target creation of a new toolchain file is the best
solution. The existing toolchain files can be used a starting point for a new
toolchain file since each one exposes the basic requirements for toolchain files
as used in the AV1 codec build.

As a temporary work around an unoptimized AV1 configuration that builds only C
and C++ sources can be produced using the following commands:

185
~~~
Tom Finegan's avatar
Tom Finegan committed
186 187
    $ cmake path/to/aom -DAOM_TARGET_CPU=generic
    $ make
188
~~~
Tom Finegan's avatar
Tom Finegan committed
189 190 191 192 193

In addition to the above it's important to note that the toolchain files
suffixed with gcc behave differently than the others. These toolchain files
attempt to obey the $CROSS environment variable.

194 195 196 197 198 199 200 201 202 203 204 205 206 207
### Sanitizers

Sanitizer integration is built-in to the CMake build system. To enable a
sanitizer, add `-DSANITIZE=<type>` to the CMake command line. For example, to
enable address sanitizer:

~~~
    $ cmake path/to/aom -DSANITIZE=address
    $ make
~~~

Sanitizers available vary by platform, target, and compiler. Consult your
compiler documentation to determine which, if any, are available.

Tom Finegan's avatar
Tom Finegan committed
208 209 210 211 212 213
### Microsoft Visual Studio builds

Building the AV1 codec library in Microsoft Visual Studio is supported. The
following example demonstrates generating projects and a solution for the
Microsoft IDE:

214
~~~
Tom Finegan's avatar
Tom Finegan committed
215 216
    # This does not require a bash shell; command.exe is fine.
    $ cmake path/to/aom -G "Visual Studio 15 2017"
217
~~~
Tom Finegan's avatar
Tom Finegan committed
218 219 220 221 222 223

### Xcode builds

Building the AV1 codec library in Xcode is supported. The following example
demonstrates generating an Xcode project:

224
~~~
Tom Finegan's avatar
Tom Finegan committed
225
    $ cmake path/to/aom -G Xcode
226
~~~
Tom Finegan's avatar
Tom Finegan committed
227

228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270
### Emscripten builds

Building the AV1 codec library with Emscripten is supported. Typically this is
used to hook into the AOMAnalyzer GUI application. These instructions focus on
using the inspector with AOMAnalyzer, but all tools can be built with
Emscripten.

It is assumed here that you have already downloaded and installed the EMSDK,
installed and activated at least one toolchain, and setup your environment
appropriately using the emsdk\_env script.

1. Download [AOMAnalyzer](https://people.xiph.org/~mbebenita/analyzer/).

2. Configure the build:

~~~
    $ cmake path/to/aom \
        -DENABLE_CCACHE=1 \
        -DAOM_TARGET_CPU=generic \
        -DENABLE_DOCS=0 \
        -DCONFIG_ACCOUNTING=1 \
        -DCONFIG_INSPECTION=1 \
        -DCONFIG_MULTITHREAD=0 \
        -DCONFIG_RUNTIME_CPU_DETECT=0 \
        -DCONFIG_UNIT_TESTS=0 \
        -DCONFIG_WEBM_IO=0 \
        -DCMAKE_TOOLCHAIN_FILE=path/to/emsdk-portable/.../Emscripten.cmake
~~~

3. Build it: run make if that's your generator of choice:

~~~
    $ make inspect
~~~

4. Run the analyzer:

~~~
    # inspect.js is in the examples sub directory of the directory in which you
    # executed cmake.
    $ path/to/AOMAnalyzer path/to/examples/inspect.js path/to/av1/input/file
~~~

271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288
### Extra build flags

Three variables allow for passing of additional flags to the build system.

- AOM\_EXTRA\_C\_FLAGS
- AOM\_EXTRA\_CXX\_FLAGS
- AOM\_EXTRA\_EXE\_LINKER\_FLAGS

The build system attempts to ensure the flags passed through the above variables
are passed to tools last in order to allow for override of default behavior.
These flags can be used, for example, to enable asserts in a release build:

~~~
    $ cmake path/to/aom \
        -DCMAKE_BUILD_TYPE=Release \
        -DAOM_EXTRA_C_FLAGS=-UNDEBUG \
        -DAOM_EXTRA_CXX_FLAGS=-UNDEBUG
~~~
Tom Finegan's avatar
Tom Finegan committed
289 290 291 292 293

## Testing the AV1 codec

### Testing basics

294 295 296
There are several methods of testing the AV1 codec. All of these methods require
the presence of the AV1 source code and a working build of the AV1 library and
applications.
Tom Finegan's avatar
Tom Finegan committed
297

298
#### 1. Unit tests:
Tom Finegan's avatar
Tom Finegan committed
299 300 301

The unit tests can be run at build time:

302
~~~
Tom Finegan's avatar
Tom Finegan committed
303 304 305 306 307 308 309
    # Before running the make command the LIBAOM_TEST_DATA_PATH environment
    # variable should be set to avoid downloading the test files to the
    # cmake build configuration directory.
    $ cmake path/to/aom
    # Note: The AV1 CMake build creates many test targets. Running make
    # with multiple jobs will speed up the test run significantly.
    $ make runtests
310 311 312
~~~

#### 2. Example tests:
Tom Finegan's avatar
Tom Finegan committed
313 314 315

The example tests require a bash shell and can be run in the following manner:

316
~~~
Tom Finegan's avatar
Tom Finegan committed
317 318 319 320 321 322 323 324
    # See the note above about LIBAOM_TEST_DATA_PATH above.
    $ cmake path/to/aom
    $ make
    # It's best to build the testdata target using many make jobs.
    # Running it like this will verify and download (if necessary)
    # one at a time, which takes a while.
    $ make testdata
    $ path/to/aom/test/examples.sh --bin-path examples
325
~~~
Tom Finegan's avatar
Tom Finegan committed
326

327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377
#### 3. Encoder tests:

When making a change to the encoder run encoder tests to confirm that your
change has a positive or negligible impact on encode quality. When running these
tests the build configuration should be changed to enable internal encoder
statistics:

~~~
    $ cmake path/to/aom -DCONFIG_INTERNAL_STATS=1
    $ make
~~~

The repository contains scripts intended to make running these tests as simple
as possible. The following example demonstrates creating a set of baseline clips
for comparison to results produced after making your change to libaom:

~~~
    # This will encode all Y4M files in the current directory using the
    # settings specified to create the encoder baseline statistical data:
    $ cd path/to/test/inputs
    # This command line assumes that run_encodes.sh, its helper script
    # best_encode.sh, and the aomenc you intend to test are all within a
    # directory in your PATH.
    $ run_encodes.sh 200 500 50 baseline
~~~

After making your change and creating the baseline clips, you'll need to run
encodes that include your change(s) to confirm that things are working as
intended:

~~~
    # This will encode all Y4M files in the current directory using the
    # settings specified to create the statistical data for your change:
    $ cd path/to/test/inputs
    # This command line assumes that run_encodes.sh, its helper script
    # best_encode.sh, and the aomenc you intend to test are all within a
    # directory in your PATH.
    $ run_encodes.sh 200 500 50 mytweak
~~~

After creating both data sets you can use `test/visual_metrics.py` to generate a
report that can be viewed in a web browser:

~~~
    $ visual_metrics.py metrics_template.html "*stt" baseline mytweak \
      > mytweak.html
~~~

You can view the report by opening mytweak.html in a web browser.


Tom Finegan's avatar
Tom Finegan committed
378 379 380 381 382 383 384 385 386
### IDE hosted tests

By default the generated projects files created by CMake will not include the
runtests and testdata rules when generating for IDEs like Microsoft Visual
Studio and Xcode. This is done to avoid intolerably long build cycles in the
IDEs-- IDE behavior is to build all targets when selecting the build project
options in MSVS and Xcode. To enable the test rules in IDEs the
`ENABLE_IDE_TEST_HOSTING` variable must be enabled at CMake generation time:

387
~~~
Tom Finegan's avatar
Tom Finegan committed
388 389 390 391
    # This example uses Xcode. To get a list of the generators
    # available, run cmake with the -G argument missing its
    # value.
    $ cmake path/to/aom -DENABLE_IDE_TEST_HOSTING=1 -G Xcode
392
~~~
Tom Finegan's avatar
Tom Finegan committed
393 394 395 396 397 398 399

### Downloading the test data

The fastest and easiest way to obtain the test data is to use CMake to generate
a build using the Unix Makefiles generator, and then to build only the testdata
rule:

400
~~~
Tom Finegan's avatar
Tom Finegan committed
401 402 403
    $ cmake path/to/aom -G "Unix Makefiles"
    # 28 is used because there are 28 test files as of this writing.
    $ make -j28 testdata
404
~~~
Tom Finegan's avatar
Tom Finegan committed
405 406 407

The above make command will only download and verify the test data.

408 409 410 411 412 413 414
### Additional test data

The test data mentioned above is strictly intended for unit testing.

Additional input data for testing the encoder can be obtained from:
https://media.xiph.org/video/derf/

415 416 417 418 419 420 421 422 423 424 425
### Sharded testing

The AV1 codec library unit tests are built upon gtest which supports sharding of
test jobs. Sharded test runs can be achieved in a couple of ways.

#### 1. Running test\_libaom directly:

~~~
   # Set the environment variable GTEST_TOTAL_SHARDS to 9 to run 10 test shards
   # (GTEST shard indexing is 0 based).
   $ export GTEST_TOTAL_SHARDS=9
426 427
   $ seq 0 $(( $GTEST_TOTAL_SHARDS - 1 )) \
       | xargs -n 1 -P 0 -I{} env GTEST_SHARD_INDEX={} ./test_libaom
428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450
~~~

To create a test shard for each CPU core available on the current system set
`GTEST_TOTAL_SHARDS` to the number of CPU cores on your system minus one.

#### 2. Running the tests via the CMake build:

~~~
    # For IDE based builds, ENABLE_IDE_TEST_HOSTING must be enabled. See
    # the IDE hosted tests section above for more information. If the IDE
    # supports building targets concurrently tests will be sharded by default.

    # For make and ninja builds the -j parameter controls the number of shards
    # at test run time. This example will run the tests using 10 shards via
    # make.
    $ make -j10 runtests
~~~

The maximum number of test targets that can run concurrently is determined by
the number of CPUs on the system where the build is configured as detected by
CMake. A system with 24 cores can run 24 test shards using a value of 24 with
the `-j` parameter. When CMake is unable to detect the number of cores 10 shards
is the default maximum value.
Tom Finegan's avatar
Tom Finegan committed
451 452 453

## Coding style

454 455 456
We are using the Google C Coding Style defined by the
[Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html).

Tom Finegan's avatar
Tom Finegan committed
457
The coding style used by this project is enforced with clang-format using the
458 459 460 461 462 463 464 465 466 467
configuration contained in the
[.clang-format](https://chromium.googlesource.com/webm/aom/+/master/.clang-format)
file in the root of the repository.

You can download clang-format using your system's package manager, or directly
from [llvm.org](http://llvm.org/releases/download.html). You can also view the
[documentation](https://clang.llvm.org/docs/ClangFormat.html) on llvm.org.
Output from clang-format varies by clang-format version, for best results your
version should match the one used on Jenkins. You can find the clang-format
version by reading the comment in the `.clang-format` file linked above.
Tom Finegan's avatar
Tom Finegan committed
468 469 470

Before pushing changes for review you can format your code with:

471
~~~
Tom Finegan's avatar
Tom Finegan committed
472 473 474
    # Apply clang-format to modified .c, .h and .cc files
    $ clang-format -i --style=file \
      $(git diff --name-only --diff-filter=ACMR '*.[hc]' '*.cc')
475
~~~
Tom Finegan's avatar
Tom Finegan committed
476 477 478 479

Check the .clang-format file for the version used to generate it if there is any
difference between your local formatting and the review system.

480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589
Some Git installations have clang-format integration. Here are some examples:

~~~
    # Apply clang-format to all staged changes:
    $ git clang-format

    # Clang format all staged and unstaged changes:
    $ git clang-format -f

    # Clang format all staged and unstaged changes interactively:
    $ git clang-format -f -p
~~~

## Submitting patches

We manage the submission of patches using the
[Gerrit](https://www.gerritcodereview.com/) code review tool. This tool
implements a workflow on top of the Git version control system to ensure that
all changes get peer reviewed and tested prior to their distribution.

### Login cookie

Browse to [AOMedia Git index](https://aomedia.googlesource.com/) and login with
your account (Gmail credentials, for example). Next, follow the
`Generate Password` Password link at the top of the page. You’ll be given
instructions for creating a cookie to use with our Git repos.

### Contributor agreement

You will be required to execute a
[contributor agreement](http://aomedia.org/license) to ensure that the AOMedia
Project has the right to distribute your changes.

### Testing your code

The testing basics are covered in the [testing section](#testing-the-av1-codec)
above.

In addition to the local tests, many more (e.g. asan, tsan, valgrind) will run
through Jenkins instances upon upload to gerrit.

### Commit message hook

Gerrit requires that each submission include a unique Change-Id. You can assign
one manually using git commit --amend, but it’s easier to automate it with the
commit-msg hook provided by Gerrit.

Copy commit-msg to the `.git/hooks` directory of your local repo. Here's an
example:

~~~
    $ curl -Lo aom/.git/hooks/commit-msg https://chromium-review.googlesource.com/tools/hooks/commit-msg

    # Next, ensure that the downloaded commit-msg script is executable:
    $ chmod u+x aom/.git/hooks/commit-msg
~~~

See the Gerrit
[documentation](https://gerrit-review.googlesource.com/Documentation/user-changeid.html)
for more information.

### Upload your change

The command line to upload your patch looks like this:

~~~
    $ git push https://aomedia-review.googlesource.com/aom HEAD:refs/for/master
~~~

### Incorporating reviewer comments

If you previously uploaded a change to Gerrit and the Approver has asked for
changes, follow these steps:

1. Edit the files to make the changes the reviewer has requested.
2. Recommit your edits using the --amend flag, for example:

~~~
   $ git commit -a --amend
~~~

3. Use the same git push command as above to upload to Gerrit again for another
   review cycle.

In general, you should not rebase your changes when doing updates in response to
review. Doing so can make it harder to follow the evolution of your change in
the diff view.

### Submitting your change

Once your change has been Approved and Verified, you can “submit” it through the
Gerrit UI. This will usually automatically rebase your change onto the branch
specified.

Sometimes this can’t be done automatically. If you run into this problem, you
must rebase your changes manually:

~~~
    $ git fetch
    $ git rebase origin/branchname
~~~

If there are any conflicts, resolve them as you normally would with Git. When
you’re done, reupload your change.

### Viewing the status of uploaded changes

To check the status of a change that you uploaded, open
[Gerrit](https://aomedia-review.googlesource.com/), sign in, and click My >
Changes.
Tom Finegan's avatar
Tom Finegan committed
590 591 592 593 594 595 596 597 598 599

## Support

This library is an open source project supported by its community. Please
please email aomediacodec@jointdevelopment.kavi.com for help.

## Bug reports

Bug reports can be filed in the Alliance for Open Media
[issue tracker](https://bugs.chromium.org/p/aomedia/issues/list).