short-circuit av1_get_tx_scale to always return zero when
CONFIG_DAALA_TX, and remove it from the actual Daala TX toplevel

This has potential overflow consequences for any metrics computation
based on pixels; as such, also force use of the high-bitdepth path in
each of these case.

subset-1:
monty-rest-of-stack-baseline-s1@2017-11-13T00:39:03.881Z ->
monty-rest-of-stack-noshift-s1@2017-11-13T14:37:42.541Z

   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.0030 | -0.0523 |  0.2656 |  -0.0239 | -0.0033 | -0.0029 |     0.0067

objective-1-fast --limit=4:
monty-rest-of-stack-baseline-o1f4@2017-11-13T00:37:06.999Z ->
monty-rest-of-stack-noshift-o1f4@2017-11-13T14:37:16.992Z

   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.0264 |  0.2303 |  0.0822 |  -0.0109 | -0.0395 | -0.0709 |     0.0538

Change-Id: I57da71861f105dc7a404fa75a75bde573855ef79

This experiment has been abandonned for AV1.

Change-Id: I3501181d096873dec35ea58e4b55530fa76e683f

Remove the previous experiment and now use the same name for a
simpler experiment that only enables 4:1 transforms for 4:1
partitions when ext_partition_types is on, and that which was
previously enabled with the USE_RECT_TX_EXT macro.

Change-Id: Iccc35744bd292abf3c187da6f23b787692d50296

Change-Id: Ic4e16f30827e2e2e2dd140aee94d309b049dd063

To faciliate using intrabc on interframes.

Change-Id: Ibfe376190adf24d15198c5fb548e1050e191a3d6

Change-Id: I96e5ff72caee8935efb7535afa3a534175bc425c

This reverts commit 060e192b.

Change-Id: I5700d351a3cbb682ec49a0efb9cca4d0e83f9a3a

Include a few RDO fixes that properly account for the extra flag
cost at the right place. Also entropy coding of the flag is made
contexted on tx_size and based on cdf framework.

Change-Id: I52f98ace7e253ccc08917a4a7b517515d1d58a98

Conveys depth from the largest transform size instead of the
actual transform size. Besides, the max depth is now limited
by the macro MAX_TX_DPETH set at 2.

Results: BDRATE lowres (30 frames): -0.005%

Change-Id: I1ccbac8ee18c77b816a6a8f500abfaa7892b21de

In preparation for Daala unified LBD/HBD TX, add (and use) is_hbd
field in TxfmPama structure.  This field indicates whether or not
pixel data is using 8 or 16 bit reference buffers (currently ambiguous
in the case of 8 bit input).

Change-Id: I28bca792a48ffa00e208617adb072b08ff816e3c

This causes no change in the encoder output.
Comparing simple SSE-based RDO with the switch to
txfm_rd_in_plane, the overhead is reduced by 23% ~ 50%.
The total encode time increase is now 2.3% ~ 3.1%.

Change-Id: I48c76216871f8ed68631815fd781697139305e94

This experiment has been cleared by Tapas.

Also, fix a couple of hash signatures in the test while we are at it.

Change-Id: I1658bcb07913cf8bd47cfffadd729e16d5c55fc3

Support cdf for jnt_comp read/write.

Change-Id: I2c29277a8b06b3e9f571355946b70ce0d492fbb2

One_sided_compound is less likely to provide gains for jnt_comp.
Stop jnt_comp search when one_sided_compound is true. It saves coding
time as well as bit overhead.

Change-Id: I6714e5d211eba14a5793bdae7215ccfcc1d8f67a

Tested on screen_content, it has very little impact on compression quality.

Change-Id: I49c7d5e8de8420c0d2af1b327c084a2667cfea40

Results on Subset 1
   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.0354 | -0.2567 | -0.3941 |   0.0104 | -0.0084 |  0.0120 |    -0.0996

https://arewecompressedyet.com/?job=master%402017-11-03T15%3A57%3A30.643Z&job=cfl-av1-DC_PRED%402017-11-03T16%3A00%3A10.866Z

BUG=aomedia:928

Change-Id: I4e26e8c56d2246ca32b8d86145ef67f6df90d8d1

The transform search set is being determined based on the square
transform size one level below the block size. MRC_DCT is not
included in the transform set for 16x16 so it will be skipped in
the transform search for 32x32 blocks. This change allows it to
be included in the search.

Change-Id: I82395790dde9288531336a56e93575071d070572

Remove the redundant handle_inter_mode calls from the jnt-comp
encoding route.

Change-Id: I1f4fded525cfd3ead7d06c977ab8d99cb7f02273

Change-Id: Ie392e88609554fea99e74c94812799f72b78725b

Generalize quantizer setup so that quantization and TX can use
different coefficient shifts/scalings without inpacting RDO lambda
generation.

This patch is documentaiton + a minor refactor setting up later
work; it causes no functional change.

monty-daalaTX-fulltest-Daalabaseline-o1f@2017-11-07T00:01:20.779Z ->
 monty-daalaTX-fulltest-DaalaRDO-o1f@2017-11-07T00:02:31.347Z

PSNR | PSNR Cb | PSNR Cr | PSNR HVS | SSIM | MS SSIM | CIEDE 2000
 N/A |  0.0000 |  0.0000 |      N/A |  N/A |     N/A |        N/A
(note-- the numbers above were collected using --cpu-used=3, which
 appears to be newly broken.)

monty-daalaTX-fulltest-Daalabaseline-o1f4@2017-11-07T05:59:16.553Z ->
 monty-daalaTX-fulltest-DaalaRDO-o1f4@2017-11-07T05:59:50.180Z

  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0000 |  0.0000 |  0.0000 |   0.0000 | 0.0000 |  0.0000 |     0.0000

monty-daalaTX-fulltest-Daalabaseline-s1@2017-11-07T00:01:46.582Z ->
 monty-daalaTX-fulltest-DaalaRDO-s1@2017-11-07T00:02:56.282Z

  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0000 |  0.0000 |  0.0000 |   0.0000 | 0.0000 |  0.0000 |     0.0000

Change-Id: Ia5a3c052eacd17184ca1b0fe6d032cfb9afdb77f

Change-Id: Ib5de3d1c1acce676b3fcc12b4c237d681ac9a845

Always assume ONE_SIDED_COMPOUND is true as it has been adopted
officially. Further, this patch also fixes the cmake setup for
the experiment of EXT_COMP_REFS, as when EXT_COMP_REFS is on, it
shall always have ONE_SIDED_COMPOUND on (but that was not the case
without this patch).

We leave EXT_COMP_REFS for the time being, even though it is also
adopted, since by keeping this flag, it may allow its turning off to
tune the encoder speed. The experiment of EXT_COMP_REFS should be
removed eventually at the closing of the bitstream.

Change-Id: I0644318e522f5b48d5bba4e4391104a24d43cdc3

Change-Id: I853a0e4b5fc7e7b5c1745b401214ef71b65aad60

DV and ref DV should not have subpel values.

Change-Id: I7c47c442936f1d6bda36314812c44498ba6195a6

Return invalid rate (previously only invalid rdcost) if the
mode combination to check is < 8x8 tx_size + filter_intra mode.

BUG=aomedia:1006

Change-Id: If90f431c7692473c88ac7a644bfa969a1acb3573

The refactoring serves two purposes:
1. Separate code paths for jnt_comp and original compound average
computation. It provides function interface for jnt_comp while leaving
original compound average computation unchanged. In near future, SIMD
functions can be added for jnt_comp using the interface.

2. Previous implementation uses a hack on second_pred. But it may cause
segmentation fault when the test clip is small. As reported in Issue
944. This refactoring removes hacking and make it possible to address
the seg fault problem in the future.

Change-Id: Idd2cb99f6c77dae03d32ccfa1f9cbed1d7eed067

This experiment has been abandonned for AV1.

Change-Id: I18cf1354df928a0614a1e58b718cd96ee7999925

Development of this experiment will be deferred to AV2.

Change-Id: I3c4615a21b59508500bed8aab0a5c54413b4f284

One-sided compound ref prediction is used only when all reference
frames are one-sided.

This patch has demonstrated an encoder speedup of ~28%.

Using the following configure setups, the coding performance has been
dropped on Google test sets (50 frames) in BDRate by ~0.2% for lowres
and by ~0.1% for midres (Corresponding performance impact should be
smaller on AWCY):
--enable-experimental --disable-convolve-round --disable-ext-partition
--disable-ext-partition-types --disable-txk-sel --disable-txm

Change-Id: I585bbffb2f8d154e8f52a1e79a84eff8bb4a471d

0.159% gain on lowres 60 frames, compared to 0.236% gain if we don't
restrict it in small tx blocks.
(--disable-ext-partition --disable-ext-partition-types
 --disable-convolve-round --disable-ext-comp-refs)

Change-Id: I1d1c5474ca27de9dec992ea30a9883afd7a56474

The first level is turned on for speed 1.

Change-Id: I3dba0f0250b97a25e174cacc2a46ca7f76572c85

Change-Id: I4270d1260854ac27b68c5694ca8102b92bee6faa

Use a neural-network-based binary classifier to predict the first split
decision on the highest level of the TX size RD search tree. Depending
on how confident we are in the prediction we either keep full unmodified
TX size search or use the largest possible TX size and stop any further
search.

Average speed-up: 3-4%
Quality loss (lowres): 0.062%
Quality loss (midres): 0.018%

Change-Id: I64c0317db74cbeddfbdf772147c43e99e275891f

Change-Id: I299e2f2a1967f867a5452e0c449abe5243ac5d13

This experiment has been adopted, we can simplify the code
by dropping the associated preprocessor conditionals.

Change-Id: I02ed47186bbc32400ee9bfadda17659d859c0ef7

This name better reflects the functionality of the mode.

Change-Id: Idc2372e7c7b226f70f63fa4f272a88cb4d857089

The speed feature simply restricts the number of depths
searched. Currently it is turned on by default for speeds>=1.
The coding efficiency impact (tested on lowres 30 frames) seems
to be ~0.15% and the speedup is in the order of 15%.

Change-Id: I514832bd7df937292875f73d9c9026e49ac576f2

USE_UNI_COMP_REFS is always true when ext-comp-refs is enabled and is
always false otherwise.

Change-Id: Ic6061a4ab88a2ebfeb11f178872ab13909060c0b

This addresses an inconsistency between the set used
to decode the tx_type in the bitstream and the set used
for the tx_type search. Previously, the set used to
read/write the tx_type was based on the smallest tx_size
in the vartx partitioning, but the search uses a set
based on the largest possible tx_size. This patch
changes the tx_type search to use the transform type
set associated with the tx_size 1 recursive level down from
the max square tx_size to make the search more consistent
with the bitstream syntax. If a tx_size is selected for an
invalid tx_type, DCT_DCT is used for that partition instead.

This patch also adds assertions to all exposed transform
functions to ensure that no illegal transform type/size
combinations occur.

This currently gets a 0.1% drop in performance on lowres.
The drop is due to the reduction of the tx_types available
for 32x16 and 16x32 transform sizes. Before this patch,
32x16 and 16x32 transforms were getting assigned a
set of 12 tx_types, some of which we did not intend to
support for these sizes.

Change-Id: I44aca4876b261c345623cd04ad6235bca4532701

Change-Id: Icbedc16b01adf9b48f0f357ec89143462865d54e