GitLab

This will simplify the range management of rect txfm

Change-Id: Icf678fe735dd299c6c42a215c592611025e87ba6

Change-Id: Ie9a331facd12a9d762ede09451f6d44b6b3ef795

Change-Id: I22d0fac0d8e94fc02a7adf13b144ed17097ff84b

Add clamping to 16/18/20 bits after first pass of the inverse
transform for 8/10/12 bit source espectively, and clampaing to 16
bits after second stage of the inverse transform.

BUG=aomedia:30

Change-Id: I7a68334c77baecb359a4fb3b801fdfd9d5a62f29

The cos bits and 2d tx algorithms are also adjusted accordingly to
meet the 32-bit limit.

Change-Id: I9048f3d3689ff1ef1bb84888ed9f43cdc4371411

Change-Id: Ib37752952832cda36c323e8adcc564b1811b563a

This patch clears all test failures with coeff range checking
enabled for forward and inverse transforms. Also this
patch ensures that there are no transposes for any of the
rectangular transforms.
Some fine-tunnig and refactoring are still pending.
Some of the tests still need to be rewritten.

Change-Id: Ib0e3a4ceccef665ba007d121f536fad7135f38d5

This will clear range check failures related to fwd square txfms
when using --enable-coefficient_range_checking.

Change-Id: I5cdc1751b979f612b33d6e00a47277a18792f175

Adjusst txfm stage ranges so that Cfg tests pass, however
tests with --enable-coefficient-range-checking are still
failing. Those are to be addressed subsequently.

Change-Id: I1b15406a525a78322211e69a0d3a796823bb645e

Change-Id: I62117adde6f403c02667903a31454b2e3cfea4aa

This is true independent of CONFIG_TXMG flag, so no need for the other
code path.

BUG=aomedia:1114

Change-Id: I572c5151ca866d9d430460fb353610540c9bf025

Usage of CPU by round_shift_array goes from 2.01% to 1.04%.
Overall encoding is slightly faster (~0.05%).

This means some of the intermediate array have to be aligned.
Also, these functions were moved to common header/source files.

BUG=aomedia:1106

Change-Id: I492c9b1f2e7339c6cb83cfe68a61218642654d1b

Also, fix this test by using 'txsize_sqr_up_map' instead of
'tx_size_sqr_map' in both code and test.

BUG=aomedia:1114

Change-Id: Id72b5e72a40f79ad32819224c99822b5a98538d9

16x64 reuses scan order of 16x32
64x16 reuses scan order of 32x16

max eob is curtailed to 512 (instead of 1024) for both.

Change-Id: Iac2145aa5e3d090009e2a2f5715caa8d84dfb2ee

Change-Id: I57e6cd7ca71e975082b1431b0cf80d080cabeb9b

Change-Id: I496c2a83b227ec88a6cd59a4c227b59eeeb0fc86

This avoids copying a whole struct on return. This is important, because
this function is called frequently: once for every block that is transformed.

Overall, aomenc speed seems to be improved slightly: by 0.15% - 0.30%.

Change-Id: If44c351e1141c0fe0458daa2262aeb4afe44de46

Reuse the existing ones to support the 64x64, 64x32 and 32x64 configs
instead.

Change-Id: Ibadc65eb03518d19146f32657ccfd6949c0071cc

- Reuse scan order
- Truncate to max eob of 32*32
- Quantization and entropy coding same as done for TX_32X32
- Reuse quantization matrices of TX_32X32

Compression performance is roughly neutral:
https://arewecompressedyet.com/?job=tx64x64_oldscans%402017-11-06T03%3A11%3A53.868Z&job=tx64x64_reusescans%402017-11-06T03%3A12%3A55.738Z

Change-Id: Ie9182c1c69a42a3c1ab4fc980abbd6000c64f179

Change-Id: I7123717b2d11bca826d650c6e6b6ae137476d541

All tests are now passing.

Change-Id: Ifc1a0f3ff69f5730722a27eed092395595127e8e

Remove the previous experiment and now use the same name for a
simpler experiment that only enables 4:1 transforms for 4:1
partitions when ext_partition_types is on, and that which was
previously enabled with the USE_RECT_TX_EXT macro.

Change-Id: Iccc35744bd292abf3c187da6f23b787692d50296

Change-Id: I484121349af182f4f9525b1c992a6e77f6da7ea9

This experiment has been adopted, we can simplify the code
by dropping the associated preprocessor conditionals.

Change-Id: I02ed47186bbc32400ee9bfadda17659d859c0ef7

Change-Id: I5bff0a68602a89ce480fec049c8b2c4bce44f6bb

Change-Id: If6e84f390c6f92e8e013056c657758ba9e407a71

Change-Id: Idf01b14bed4701ce84fa1c127e01560f4764fadb

Change-Id: Ic843e99bd9b79cb9a0a26b95e3a48717ff2ec2a5

Change-Id: Ifa983d83a509cdfad78f6400df7d60c8f5b4f68c

The SSE4 function does not support 64x64 size, and was triggering an
assertion failure lowbitdepth is disabled.

BUG=aomedia:672

Change-Id: Id14e76b5c180a211a84c2e933b07e8acf72dddbc

Change-Id: Ie2f83f2f9369a22b70150ba44ddb6f82d6b6b514

Change-Id: I41a78f5066b96f59ab8f587bc5b1955f6806b103

Responding to some left over cosmetic comments from
2b5cdb1cf87c933331a16cc0221455d0a8c255e1

Change-Id: I42e126593526cedd6675adf35b9c1df78e1ddf54

Bug introduced in change: I34cdeaed2461ed7942364147cef10d7d21e3779c

BUG=aomedia:591

Change-Id: I49b9edd2bf5a482b5afea5d83d56e04a0086f797

This unifies the codepath for high-bitdepth transforms and deletes
all calls to the old deprecated versions. This required reworking
the way 1d configurations are combined in order to support rectangular
transforms.

There is one remaining codepath that calls the deprecated 4x4 hbd
transform from encoder/encodemb.c. I need to take a closer look
at what is happening there and will leave that for a followup
since this change has already gotten so large.

lowres 10 bit: -0.035%
lowres 12 bit: 0.021%

BUG=aomedia:524

Change-Id: I34cdeaed2461ed7942364147cef10d7d21e3779c

cb4x4 itself should not require these sizes.

This simplifies compatibility with other experiments, since we can
first make them work with cb4x4 (which is now on by default), and
then worry about chroma_2x2 (which is not) in separate steps.

Encoder and decoder output should remain unchanged.

Change-Id: I4e9fcdae49f238b5099a3c74a398fe993c2545f8

This adds the proper cfgs to av1_{inv/fwd}_txfm1d_cfg for
the identity transform so all hbd transforms can use
the same codepath. This has no impact on performance
since the new identity transforms that correspond with
the cfgs are not yet being called. Once this is checked in,
we should be able to delete all deprecated transform functions
and have a single code flow for all hbd transforms.

BUG=aomedia:524

Change-Id: I3d1bfbc8bc29b367e8ddf7dcd27525af0bd31067

The hbd transform configurations were originally written for all possible
2d transforms. Now that there are many more possible 2d transforms
due to EXT_TX and RECT_TX, it is simpler to write the cfg for the
4 1D transform types and compose them to make all new possible transform
types. This will allow for an easier integration of the identity transform
for EXT_TX and rectangular transforms for RECT_TX into the current
hbd transform codepath and facilitate the removal of obsolete transforms.
This has no impact on performance.

BUG=aomedia:524

Change-Id: I1e217bcd217fd637b1df94fae62d9c59a0523c1a

Change-Id: Ibd7091d444cfb3095ddb938100704d3b75cf1af4

This reverts commit f9d77bd5.

Reason for revert: 8x8 transform failures

BUG=https://bugs.chromium.org/p/aomedia/issues/detail?id=502

Change-Id: I2f6c10bc576a966bd5a878b7ee8389074bf45014