A lot of local functions have av1_ prefix due to legacy reasons.
Make them static and remove their prefix.

Change-Id: I069a3a1af583d5d307cdc56819901d1f9e004d07

In the old code, the function of allocate_gf_group_bits() in
firstpass.c handles both GF group structure design as well as the
corresponding bit allocation for each coded frame. This CL separates
the two functionalities.

It has been verified that no coding performance has been incurred by
this CL.

Change-Id: I544b4fcdc8cb37ec810373f1f820dbd2e400c39b

Change-Id: Ie67266538db41c6aa7d79a5c270c26c021583c48

Change-Id: I5cf978071fbb55040d2be88f627b600484988520

Change-Id: Idbde50deab816b669ca362e3e739c76a7f434f2c

Remove the token prob tables and counters.

Change-Id: Ic63d52d80bb922fc10b586c27a20f2378618168c

Reduce the encoder memory footprint in lv-map by ~800X.

BUG=aomedia:713

Change-Id: I7aad604cf3186a0d3c6d5b6d1aa9489ac7c36093

This reverts commit b855b8e8.

Reason for revert:
breaks the rate assumptions in:
AV1/DatarateTestLarge.BasicRateTargetingVBR
AV1/DatarateTestLarge.ChangingDropFrameThresh/2

Change-Id: I92c4709b6c1bd98a78fe5aa6f0167a534fed5997

Changes:
- Add block size define.
- Set default to mean filtering.
- Add flag in AV1_COMP for bgsprite.
- Determine to use bgsprite or not based on firstpass metrics in
define_gf_group.
- Add outlier removal for bgsprite mean blending (off by default)

Results:
lowres: -0.009 avg_psnr, -0.029 ovr_psnr, -0.102 ssim

Clips with maximum gain:
- brdige_far_cif: -2.071 avg_psnr, -2.409 ovr_psnr, -1.977 ssim
- deadline_cif: -0.148 avg_psnr, -0.137 ovr_psnr, -0.113 ssim
- brdige_close_cif: +1.741 avg_psnr, -0.258 ovr_psnr, -2.534 ssim

Change-Id: I809406020f7786e49cc80329511e22d25379d7a2

Update cdf after each superblock being finalized, and re-compute
symbol costs. Affected symbols include prediction parameters,
tx type, and new motion vector.
BDRate: -0.121% lowres, -0.089% midres

Change-Id: I005ef382762d6e3423f933cca0a0b7b40288e8bf

The change is purely an encoder-side tuning, which constrains a GF
group interval to be either 8, or 12, or a number larger than 12
(maximum of 16). The change is under ext_refs.

This work has been part of the summer internship effort by chendixi@.

Further, this CL removes the experiment of flex_refs and merges it
into ext_refs, as flex_refs has been completely focused on the encoder
side optimization for ext_refs.

On Google test sets, above (stillness + interval enforcement) achieve
a coding gain in BDRate as follows, compared against the AV1 baseline
with all tools enabled by default (excluding ext-tx, global-motion,
and convolve-round for speed concern):

lowres: avg_psnr -0.193%; ovr_psnr -0.370%; ssim -0.332%
midres: avg_psnr -0.556%; ovr_psnr -0.680%; ssim -0.798%

Change-Id: Ia992d8ef0d82cc0ef2caba5eec9b26f0358e6bde

Avoid changing the public API based on an experiment flag:

- Remove aom_config.h include from aom_image.h.
- Unconditionally include the existing API additions.
- Add some missing doc comments (silences Doxygen warnings).
- Return an error when using ctrl flags that require
  CONFIG_COLORSPACE_HEADERS when it's not enabled.
- Move colorspace_headers to the correct section of configure.
- Move CONFIG_COLORSPACE_HEADERS to the correct section of
  aom_config_defaults.cmake.
- clang-format style check appeasement

Change-Id: I7b5d72c6f9f1a5561409d7813ba59180d98d8805

Added 'cdef-dist' and 'daala-dist' options to --tune-metric'.
(Actual bodies of the options will be added in later separate commits)

Also, if --tune=cdef-dist or --tune=daala-dist,
enable-dist-8x8 is automatically turned on.

This commit is a part of prep-work to remove DIST_8X8, CDEF_DIST,
and DAALA_DIST experimental flags.

Change-Id: I0790b9537dfaf166cb59741997ae27da10697a33

Change-Id: I61165218afefe7fc59ad06a0abe0180754cdf7bd

This experiment is now adopted as it was cleared by Tapas.

Note: Palette use can still be controlled by command-line option
"--tune-content=..." in 'aomenc'.

Change-Id: I832f49f20f60c34bdef5b424755849c496687e87

Change-Id: Iec7969ffd8f53ca2f4eefd1d757cfec7b3bde131

This CL adds the use of ALTREF2_FRAME to both single / comp reference
prediction at the encoder side. In particular, the encoder keeps the
distant altref as ALTREF, and uses the internal extra altrefs to
refresh ALTREF2.

Compared with the baseline (ext_tx and global_motion disabled simply
for speed concern):
(a) lowres: avg_psnr -0.395% ovr_psnr -0.393% ssim -0.329%
(b) midres: avg_psnr -0.419% ovr_psnr -0.431% ssim -0.444%
(c) AWCY High Latency:
   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.6661 | -0.5988 | -0.6669 |  -0.6993 | -0.6988 | -0.7303 | -0.6051
(d) AWCY Low Latency:
  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0720 | -0.0505 |  0.1501 |   0.0670 | 0.0842 |  0.0517 |     0.0158

TODO list:
(1) To have altref2 incorporated with ext-comp-refs;
(2) To have altref2 fully work with new-multisymbol;
(3) To re-collect the initial default probs/cdfs;
(4) To tune the encoder gf group structure design for altref2.

Change-Id: I6ad63fd65afa903d3bba20acdb68e3b67acf7fdf

This stores frame contexts alongside a reference frame, and always
uses the frame in reference slot 0 (LAST_FRAME) as the source of
the frame context.

The encoder could then reorder reference frames as to control
which frame context is used, however currently it does not.

Low Latency AWCY result:

   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.1438 |  0.4161 |     N/A |   0.0386 | -0.0281 |  0.0453 |     0.2514

https://arewecompressedyet.com/?job=before-frame-context-signaling%402017-06-07T23%3A20%3A49.473Z&job=after-frame-context-signaling%402017-06-07T23%3A21%3A36.117Z

Change-Id: I4f6f9b12cb403573efbf9e5c3077d62f5dedc467

It is a refactoring patch, which aims to make the code ready for
implementation of in-frame mode cost update in RDO.
Also add mode cost update per sb row, but it won't affect coding
results because cdf update in RDO is not there.
Mode cost arrays are moved to MACROBLOCK because in multi-thread
coding, threads share the same AV1_COMP.

This patch does not have impact on coding results.

Change-Id: I2e8f7d7d066b23ebfbfc998269023781f359a6ff

A separate prediction mode struct is added to allow
for uv-only modes (like CfL). Note: CfL will be
added as a separate mode in an upcoming commit.

Results on Subset1 (Compared to 4266a7ed with CfL enabled)
  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0000 |  0.0000 |  0.0000 |   0.0000 | 0.0000 |  0.0000 |     0.0000

Change-Id: Ie80711c641c97f745daac899eadce6201ed97fcc

Use three metrics to identify the still gf group.
Performance:
lowres: pamphlet_cif -1.395; bowing_cif -0.989;
        others remain same. Overall -0.064
midres: snow_mnt_480p -0.827. others remain same.
        Overall -0.028

Change-Id: I22a6429c7ebdad2c36ec73c7a69cabc07e8208b7

Tile groups are now an integral part of the codec.

Change-Id: I620a88ec7a44b057d5cce0bf6cf602822a3339a9

Screen content tools can be forced on or inferred by color counting.
This prevents the inferred status of the tools from getting clobbered
when controls are evaluated.

BUG=aomedia:656

This also fixes the build with --disable-palette --enable-intrabc

Change-Id: If815a1eff2f9419117477f14cda143dd669c87c6

This reverts commit 1b5328e0.

This commit has some pretty negative consequences with block copy. I
need to be able to force block copy on for arbitrary sequences while it
is still in development, and I need to be able to keep the palette
enable status between block copy and non-block copy runs.

Change If815a1eff2f9419117477f14cda143dd669c87c6 will prevent
allow_screen_content_tools from getting clobbered and still allow us to
force it on.
BUG=aomedia:656

Change-Id: I7a2f4207b5d2da35aec7870250e8b330c58a0e13

Always let the encoder decide if "allow_screen_content_tools" should be set.

-The screen content detector(is_screen_content()) works fairly well.

-Keeping the config control may lead to mismatches. For example, user
may encode the 1st frame with the flag on, then turn the flag off for
the 2nd frame via config control. On the decoder side, the flag is on
for both frames, because the flag is only transmitted in the header of
all-intra frame, i.e. the 1st frame in this example.

BUT=aomedia:656

Change-Id: I0652fe2b9f8df19bfbf7ed7d3513205c80ea9c61

Added a 1-bit flag 'large_scale_tile'. If it is 0 that is the default value,
use normal tile coding in TILE_GROUPS. If it is 1, use large-scale tile
coding in EXT_TILE.

At large_scale_tile=1 case, if single-tile-decoding is required, then the
loopfilter is disabled.

Related API and unit tests were modified.

Change-Id: I3ba12dc3d80ccf1ab21543ab3b16c02282c34e3b

Add a new default motion_mode_cdf supporting this
experiment.
Also define the costs for the ncobmc mode tokens.

Change-Id: Iebab044e4a34e3d2a94c18893107d82055cc9c77

Turn "allow_screen_content_tools" on when the source video has many blocks
with only few different colors. The automatic detection is enabled by
defualt (or with command line flag "--tune-content=default"). With
"--tune-content=screen", the screen content tools are always turned on.

On the screen_content test set, the "default" setting is less than 0.3%
worse than the "screen" setting on keyframe encoding.

Change-Id: Iac7ab8952c96531d1fae84da1823291f5987519c

This patch adds support for 4:1 rectangular blocks to various common
data arrays, and adds new partition types to the EXT_PARTITION_TYPES
experiment which will use them.

This patch has the following restrictions, which can be lifted in
future patches:

  * ext-partition-types is incompatible with fp_mb_stats and supertx
    for the moment

  * Currently only 32x32 superblocks can use the new partition types

There's a slightly odd restriction about when we allow
PARTITION_HORZ_4 or PARTITION_VERT_4. Since these both live in the
EXT_PARTITION_TYPES CDF, read_partition() can only return them if both
has_rows and has_cols is true. This means that at least half of the
width and height of the block must be visible. It might be nice to
relax that restriction but that would imply a change to how we encode
partition types, which seems already to be in a state of flux, so
maybe it's better to wait until that has settled down.

Change-Id: Id7fc3fd0f762f35f63b3d3e3bf4e07c245c7b4fa

Removing these flags make the EC_ADAPT experiment an integral part of
 the draft AV1 bitstream definition
This commit has no effect on metrics.

Change-Id: Ice78520935e8bfa9d25cf4b8384a1b872069d09c

They do not handle border extension correctly (interpolation and
border extension do not commute unless you upsample into the
border), nor do they handle crop dimensions that are not a multiple
of 8 (the upsampled version is not sufficiently large), in addition
to using massive amounts of memory and being a criminal waste of
cache (1 byte used for every 8 bytes fetched).

This commit reimplements use_upsampled_references by computing the
subpixel samples on the fly. This implementation not only corrects
the border handling, but is also faster, while maintaining the
same quality.

HL AWCY results are basically noise:
    PSNR | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
  0.0188 |   0.0187 | 0.0045 |  0.0063 |     0.0228

Change-Id: I7527db9f83b87a7bb8b35342f7e6457cd0bef9cd

Adds a separate scale numerator for resize and superres scaling on
keyframes. They will only use this factor in their respective FIXED
modes.

Change-Id: I4ef9a5760a5423ec632d644e6c7fec674bbb46f4

This patch will not cause any performance change regardless of whether
speed_refs is on or off.

This coding tool is targeted to speed up the encoder side reference
frame selection process. The essential idea is to have two scanning
passes for each superblock of size 64x64 and this CL lays out the
initial framework but no reference frame selection is done yet:

First scanning pass - To simplify the partition and the mode
candidates (e.g. considering nearestmv / nearmv / zeromv only) and
identify the best reference frame prediction candidates;

Second scanning pass - Use the best reference frame candidate(s)
obtained from the first pass to encode the current superblock.

Change-Id: I11266d468de3077271a5e866eebd341a8014d136

Fix a bug where the encoder sets reference_mode =
COMPOUND_REFERENCE even if the frame contains some
4x4 blocks (but no larger blocks) that are single
reference. This occurs only in corner cases,
typically for very small frame sizes (e.g. 16x8).

BUG=aomedia:575

Change-Id: I5caedaa753a56e9f9b48995c1d2e0a0886aa4366

Unifies the command line arguments for resize and superes.
Removes resize arguments that are currently unused.

Change-Id: I1362dffceb8257cee539f92679d1e8d79df08e66

Adds "superres-mode" and "superres-numerator" arguments to configure
superres scaling.

There are only two modes right now: SUPERRES_FIXED and SUPERRES_DYNAMIC.
0 sets the fixed mode and 1 sets the dynamic mode.

For the fixed mode superres will scale to the provided numerator where
possible. The cumulative scale change over resizing and superres is
limited to 1/2, so it may have to back off.

For the dynamic mode, the provided numerator will be used as the initial
numerator. Asides from that it's just random for now.

Change-Id: Ibae01c77abba2599fbf51096459d09ddca458d4f

Change-Id: If93f9e41c25a5885b5927e66a298c129bb844e18

Now the single ref comp mode should work with WEDGE and
COMPOUND_SEGMENT. For motion_var, the OBMC_CAUSAL mode uses the 2nd
predictor if the neighboring block is single ref comp mode predicted.

This patch removes the mode of SR_NEAREST_NEWMV and leaves four
single ref comp modes in total:

SR_NEAREST_NEARMV
SR_NEAR_NEWMV
SR_ZERO_NEWMV
SR_NEW_NEWMV

Change-Id: If6140455771f0f1a3b947766eccf82f23cc6b67a

This patch implements the post-encode and post-decode upscaling for the
frame superresolution experiment to work.

Upscaling happens after cdef and before loop restoration.

For now, this patch forces on random-superres.

The patch also cleans up some broken rate control hooks from VP9
days, to be brought back later when the resize and superres tools
are stable.

Change-Id: If0a8f69224dfaa0f4ae7703bd429ea2af953c7a6

Move the large stack allocation from stack initialization to
dedicated mem space. This resolves the extremely large stack issue
when ext-partition, motion-var, and high bit-depth are all turned
on.

BUG=aomedia:415

Change-Id: I85b77bbc6429093fcb0152176d9e237087d6bbd8