Following the cb4x4 design, the logic related to sub8x8 handling
can be simplifed.

Change-Id: I1ed0c0327a4498614408320a583ad37607bde4af

Filter intra mode is disabled if any dimention of tx_size is
greater than 32. In this patch, the max qualified tx_size is
found and set for testing filter_intra in intra_model_yrd().

BUG=aomedia:1058

Change-Id: I105aa68e722fce74b8828860a40040764cf20118

Change-Id: I7ce37b2e43b4607c77515d802a6ad330047fc4c2

Change-Id: I3e6b13dcb8e5b5cf6ff4ba8010fe65e43336d33f

Measure the time spending on av1_cost_coeffs by turning on this
flag

Change-Id: Ide4171182d87bd52f00fe01abefc77cb2d730834

The get_y_mode function, is superfluous, not used consistently, and requires a
useless block_idx parameter than gets pass around a lot inside the
codebase for no apparent reason.

The block parameter is misleading, as it could cause people to think all these
functions actually use this value.

Change-Id: I7ae0a8d1282c009b9114c83771cce10f5c2ee397

This prepares the way for a heuristic to truncate the search.

For a given joint sign value, each plane's alpha is coded
independently and so the alpha choice is also independent.
Rather than iterating over all possible pairs, first choose
for each plane separately and then merge the choices.
Merge with RD collection and continuously update the bound.

On subset1, then encoder output is identical for most sequences:
  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0002 | -0.0058 | -0.0016 |   0.0020 | 0.0025 |  0.0040 |    -0.0022

Change-Id: Ie7d1e1622d46eaa8cf42f50c9a6bec6e59bdbc39

Based on the HW Subgroup call of December 4th 2017, we limit luma partition to
32X32.

Regression on Subset 1
  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
0.0881 |  1.3504 |  1.2936 |   0.0572 |  0.0182 |  0.0227 |     0.5204

https://two.arewecompressedyet.com/?job=CfL-PartU%402017-12-12T15%3A39%3A36.794Z&job=CfL-Max32x32%402017-12-12T16%3A10%3A09.989Z

Change-Id: I7e3cfd68097c0bc24b1426348b5fd574c4f638a0

To avoid a cascade of encodes when performing CfL RDO,
we compute DC_PRED on the partition unit. To do so, we
change the tx_size of CfL to match the size of the
partition unit (i.e. CfL partitions only contain 1
transform block).

This change requires disabling CfL when a chroma
partition-unit-sized DC_PRED is unavailable
(i.e. 4:1, 1:4 partitions and chroma partitions > 32X32).

Results on Subset1 (compared to disabling 4:1 and 1:4 partU):
   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.1243 | -1.9286 | -2.0140 |  -0.1514 | -0.1512 | -0.1947 |    -0.8066

https://two.arewecompressedyet.com/?job=master%402017-12-12T14%3A53%3A01.451Z&job=CfL-PartU%402017-12-12T15%3A39%3A36.794Z

Change-Id: I2a4adde79c10089130775b8e0df5f9c198855cad

Account for the txb_optimize effect on level map context modeling
in the transform block rate distortion optimization search.
Properly update such information in the txb hash rd cost.

BUG=aomedia:1118

Change-Id: I16cf80c57fa37929f94291a7500d84f9e0778400

Previously the tx block order, i.e. the child index, was wrong for
rectangular partition blocks.

Re-eanbles tx rd hash for rectangular blocks, encoder speed up by 7%; no
compression quality loss.

BUG=aomedia:1106

Change-Id: I615b8c56744075088943d372e607c18795cedac4

Make sure the ref dv is a valid dv.

Improves compression by about 0.3% on screen_content keyframes.

Change-Id: I3a20c1a04b0ebcad610276f21b410dcfc8ba4c4d

BUG=aomedia:1112

Change-Id: I130bc2087c32bd2d9087bf0dfe5919226f5fafa1

This patch fixes up av1_tx_type_cost to match the code in
av1_write_tx_type. Beforehand, we wrongly assumed a 32x16 block needed
to signal its transform size (with rect-tx-ext & rect-tx-ext-intra)
because we were passing 16x16 to get_ext_tx_types.

I've also changed av1_write_tx_type to use get_min_tx_size rather than
inlining its body. No functional change, but it's probably better to
use the same helper function both times.

Change-Id: Iff6ee0bff2d332d5270fe0219db88c95e0b051d0

Remove unnecessary inter / intra condition check.

Change-Id: I51b776a3df6ce45b31fe3d2f43fbf6ca7b1d14fb

We should be selecting a transform size when tx_mode is TX_MODE_SELECT
and we're not lossless. (In lossless mode, the transform size is
forced to be TX_4X4).

Change-Id: I65b346fb2116689598e2251e97897be91f481ceb

The change in encodeframe.c has no effect (it's the same test), but
the change in rdopt.c fixes a miscalculated rate for (say) 8x4 blocks
which must signal whether to use TX_8X4 or TX_4X4.

Change-Id: I1dd2556bb90e86e690dd1fabaad278813ba73ea6

Change-Id: Ia069373be864cae7a283bce3d12ee10270e6beb3

This patch enables the skip prediction feature for blocks larger than
16x16; but the tx size used in this module is still limited to 16x16.

Also makes it exit early upon large quantized coeff.

Test results:
-encoding speed up by 0~17%, on average 8.6%, based on several lowres
test sequences;
-no quality loss observed, encoding 30 frames on lowres and midres.

BUG=aomedia:1106

Change-Id: Ic2051a978eda5c8ce6a795a9d97d8d114c6edfe2

Fix the block context model used in the hash rd for lv-map.

Change-Id: Id590413bf8282b820ceca8b61132094df93be892

I had done only a partial fix before. But updating the costs seems to
make an even bigger improvement to the baseline.

BUG=aomedia:1094

Change-Id: Id427819507143c05b4dc3d4c05e6b2a66d17899f

This patch updates also the default cdfs for skip mode. It also remove
the unneeded checking on whether the best obtained RD mode aligned
with the skip mode.

For 30 frames, on both AWCY and Google lowres/midres test sets, with
the following config, the coding gain ranges from ~-0.15% to -0.23%:

--disable-ext-partition --disable-ext-partition-types --disable-txmg
--enable-jnt-comp --enable-mfmv

Change-Id: I39aeeb9e7b27ca679c1b3f9c0eea67b5e1f694e2

BUG=aomedia:1093

Change-Id: I0871c67d9a2ddc6aa9d3084f645305f6b74a999f

Change-Id: Idac0e6f4928e18c115c1055ca43f5b817ca51518

Also, splits the sub_tx_size_map array into inter and intra
in order to enable the new 4:1 transforms for inter and
intra separately.
Includes refactoring such as removing the intra_tx_size_cat_lookup
array since it is unnecessary, and consolidating the
max_txsize_rect_lookup array for convenience.

Change-Id: I112553bab612dafb973611c87f36a43e1ac4be85

Change-Id: Ia5b5685fe22293767d0e1d5c1747a33ba9d79e0a

Currently 4:1 transforms have max 2 split levels:
4:1 -> 1:1 -> 0.5:0.5.

This refactor enables split levels:
4:1 -> 2:1 -> 1:1,

by simply changing the tables in common_data.h.

The actual switch will be made in a subsequent patch.

Change-Id: I33f8d9ca5159ba3e7d02ced449ddf6f804a8f12a

Limit the prediction residual hash table access within the same
tile. This resolves a data race issue across tiles in multi-threading
encoding that triggers instable encoding results.

BUG=aomedia:1088

Change-Id: Ia4a415a0750bd60ee8ac4e56aa1cd39ec99e19c7

Change-Id: I09cf970a9be3a7dfe7e134bec5d778cda77f2682

Exit early on large prediction error. This reduces the overhead of the
tx skip prediction, especially for low quantizers.

BUG=aomedia:1106

Change-Id: Icad8e01cdeb2e8f4cf0befa7f5a89e088f3c17e5

This simplifies the profile lookup table and removes the requirement
of computing the ctx for new-quant. This change produces a
0.01% drop in performance on 40 frames of lowres.

Change-Id: Ifcbd7d6e427090e02bb43f6de15e551796aa74c4

In addition to k-means, consider using the dominant colors directly.

Improve keyframe by about 1% on the screen_content testset.

Change-Id: I08a932c322cfe36fb8def778d14f96d71c1017db

This experiment has been abandonned for AV1.

Change-Id: I3a6dfd2178aff2502d0ea7e9dd6d584b4db0eb1f

This experiment allows two choices for the frame-level reference
mode flag: SINGLE_REFERENCE and REFERENCE_MODE_SELECT. It removes
the choice of COMPOUND_REFERENCE, as it has been barely used.

Change-Id: I8af18acd2fe3c0d4928f2b05f35aad0ebcb1556a

Moving CfL to using partition unit DC_PRED requires 4:1 and 1:4 DC_PRED,
which are not currently implemented. A simple solution is to disable CfL
for 4:1 and 1:4 partitions.

CfL is also disabled for luma intra partitions < 4x4. This is inherent
to luma intra prediction partition sizes. We add an assert to enforce
this.

Resulting in the following regression for Subset1
   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
-0.0093 |  0.1803 |  0.1519 |  -0.0180 | 0.0256 |  0.0226 |     0.0352

https://two.arewecompressedyet.com/?job=CfL%402017-11-30T19%3A05%3A05.639Z&job=CfL-Disable-4to1%402017-11-30T19%3A04%3A00.761Z

Change-Id: Ie2c8b4d9cb6b6f33a103b540209e1a2fb6df74a7

This is for speeding up the testing process

Change-Id: I90866fa239794f14e4801675d471dbf50b779d18

Extend context model for comp_group_idx.
Save sending comp_group_idx when masked_compound is not allowed.

Change-Id: Ia7ae53958c9e1c8fe07be4b14a425d9b8648082d

Remove COMPOUND_AVERAGE from compound_type_cdfs since it is now grouped
to compound_idx. However, COMPOUND_AVERAGE is still used elsewhere.

Change-Id: Ie0d460aabf9252e80eb4130cfef9aaf0efc3969d

Drop unnecessary ref_mvs assignment steps in the rate-distortion
optimization process.

Change-Id: Icbd0436d2f962ec6b1141b889cf5892b863f081d

This patch fixes and enables rectangular intra transform
sizes for 4:1 partitions (that were turned off before).
4:1 partitions can now use rectangular intra predictions with
2:1 rectangular transform sizes.
BDRATE lowres (single keyframe): -0.612%

Change-Id: I6f062f7c08aae8eeb0a55d31e792c8f7e3f302a2