Change-Id: I0c689fb44e9e91539c450d26672d7a1d92e92900

This fixes edge extension if SB [0,1] doesn't use B_PRED but [1,1] does.

Change-Id: I48354a9dba0de16472938824f5e6db87ce61979f

Change-Id: I87d571008c73f0a8514e0a864405aadb82fd1bc0

Stray CONFIG_HYBRIDTRANSFORM caused decode mismatch.

Change-Id: I1d77bad9b366f7a6ceb3cb45269ed5badb5c4c53

Refactor per-transform copy & paste into a common function
update_coef_probs_common() and read_coef_probs_common(). The dry-run and
bit-writing loops in the encoder are still obvious candidates to be made
common, but they start to diverge a bit in the next commit, so are left
as-is for now.

Change-Id: I896bd3f4a073a6296eab7e92463fef79d8c6c08c

There is a macro DEFAULT_INTERP_FILTER defined in encoder/onyx_if.c that
is set as EIGHTTAP for now - so SWITCHABLE is not really used. Ideally,
this should be SWITCHABLE but that would make the encoder quite a bit slower.
We will change the default filter to SWITCHABLE once we find a faster way to
search for switchable filters.

Change-Id: Iee91832cdc07e6e14108d9b543130fdd12fc9874

Change-Id: I0d306b0062be59bc785b65882691c956c8b5e7c8

Removes the old mv encoding code.

Change-Id: I1026d48cc5ac15ccb776f98463e929254c3dc7da

Change-Id: I99f1982b30a630a9a070a8326d83b34a33cba14c

For splitmv, where partitioning is 8x16, 16x8 or 8x8, this patch
uses the 8x8 transform (instead of the 4x4) if txfm_mode is
ALLOW_8X8 or ALLOW_16X16. For TX_MODE_SELECT, splitmv can indicate
which of the 2 transform sizes (4x4 or 8x8) it wants to use.

Gains (with hybridtx4x4/8x8/16x16 and tx_select experiments
enabled) on derf: +0.9%, HD: +0.4%, STD/HD: +0.8% (SSIM or overall
PSNR, both metrics show similar improvements).

Change-Id: Ide954b000b415548ed92a7ac78e24f36e60fcf06

Allows B_VL_PRED & B_LD_PRED modes to be used for all blocks
within a MB in B_PRED mode. These modes were temporarily
disabled with super-block coding.

Change-Id: I973b9bdb82c3da5f12d7cc963162a28805f25303

Pass the bool coder to be used explicitly. This avoids cases where two
different bool coders can be addressed from the same function. Also be
more consistent with bool coder variable naming, start to standardize
on 'bc'.

Change-Id: I1c95e2fdbe24ebe8c0f84924daa1728e3b054a31

Separates the logic on transform type selection previously spread out
over a number of files into a separate function. Currently the tx_type
field in b_mode_info is not used, but still left in there to eventually
use for signaling the transform type in the bitstream.

Also, now for tx_type = DCT_DCT, the regular integer DCT is used, as
opposed to the floating point DCT used in conjuction with hybrid
transform.

Results change somewhat due to the transform change, but are within
reasonable limits. The hd/std-hd sets are slightly up, while derf/yt
are slightly down.

Change-Id: I5776840c2239ca2da31ca6cfd7fd1148dc5f9e0f

remove useless space after address-of operator.

Change-Id: I1fb9e82e8d6cf87558fbd454fb5c0f87599ca2ab

Packs the bitstream with each mb's residual following its mode/mv
information.

TODO: There are still a few fields that should be packed into partition
0 but are included in partition 1, due to them being serialized from
write_kfmodes/pack_inter_mode_mvs, which execute after the first
partition is finalized. These need to be separated out into a separate
function, similar to mb_mode_mv_init() in decodemv.c.

Change-Id: I43a46c363601ab36954d07ebe498760e1e2e3af4

Rather than decoding all modes/mvs separately, decode them per MB. This
forces the mode which was already used form the CONFIG_NEWBESTREFMV and
CONFIG_SUPERBLOCKS experiments, and is a precursor to changing to
interleaved encoding.

Change-Id: If19ee74ac8a987846d1cd0cf2b2e02a82f1a43ad

This reinstates reverted commit 2113a831

Change-Id: I9a9af13497d1e58d4f467e3e083fddf06b1b786c

This reverts commit 2113a831

Code clean up - removed rtcd

Change-Id: Id963ecf53c370b1d99484ef18d6befeed7e0c748

This allowed decoding from any key frame after skipping the frames
prior.

Change-Id: I096fccad5346d75ae50a017c94eb8d772a7e8b00

The update_mb_segmentation_map flag was being signalled earlier than
other data dependent on that flag. Consolidate this data so it's
parsed within the same if-scope as the flag is originally parsed in.

Change-Id: I10e90b4f511856445ef75a85a44ff441e1e5e672

Change-Id: Iecb282fc89f9b5145ef31c5eda294ad42bc32a5d

Change-Id: I22aa803ffff330622cdb77277e7b196a9766f882

Results: derf (vanilla or +hybridtx) +0.2% and (+hybrid16x16
or +tx16x16) +0.7%-0.8%; HD (vanilla or +hybridtx) +0.1-0.2%
and (+hybrid16x16 or +tx16x16) +1.4%, STD/HD (vanilla or +hybridtx)
about even, and (+hybrid16x16 or +tx16x16) +0.8-1.0%.

Change-Id: I03899e2f7a64e725a863f32e55366035ba77aa62

A variety of issues related to tangling of various hybrid transform
experiments.

Change-Id: I84dac65ff513f8b024407760df55fffa95e198de

Separates the entropy coding context models for 4x4, 8x8 and 16x16
ADST variants.

There is a small improvement for HD (hd/std-hd) by about 0.1-0.2%.
Results on derf/yt are about the same, probably because there is not
enough statistics.

Results may improve somewhat once the initial probability tables are
updated for the hybrid transforms which is coming soon.

Change-Id: Ic7c0c62dacc68ef551054fdb575be8b8507d32a8

With this change, even if hybridtransform8x8 experiment is off,
8x8 dct is used for the I8x8 mode. However note that the gains
observed with the hybridtransform8x8 experiment will now be less,
since part of the gain is now merged in.

Change-Id: I9afb3880906fd0a1368a374041fc08efcf060c54

for MBs using i8x8 prediciton mode when hybridtransform8x8 is not
enabled in configure.

Change-Id: I3e3c8bb1c0261108c1c71213ca9557b0ce558036

Some cleanups on the transform size and type selection logic.

Change-Id: If2e9675459482242cf83b4f7de7634505e3f6dac

This reverts commit c5af82b7

There was a mismatch bug introduced by this patch. Hence reverting until it is fixed.

Some cleanups that will make it easier to maintain the code
and incorporate upcoming changes on  entropy coding for the
hybrid transforms.

Change-Id: I44bdba368f7b8bf203161d7a6d3b1fc2c9e21a8f

Adds a new experiment with redesigned/refactored motion vector entropy
coding. The patch also takes a first step towards separating the
integer and fractional pel components of a MV. However the fractional
pel encoding still depends on the integer pel part and so they are
not fully independent. Further experiments are in progress to see
how much they can be decoupled without affecting performance.
All components including entropy coding/decoding, costing for MV
search, forward updates and backward updates to probability tables,
have been implemented.

Results so far:
derf: +0.19%
std-hd: +0.28%
yt: +0.80%
hd: +1.15%

Patch: Simplifies the fractional pel models:
derf: +0.284%
std-hd: +0.289%
yt: +0.849%
hd: +1.254%

Patch: Some changes in the models, rebased.
derf: +0.330%
std-hd: +0.306%
yt: +0.816%
hd: +1.225%

Change-Id: I646b3c48f3587f4cc909639b78c3798da6402678

Enable ADST/DCT of dimension 16x16 for I16X16 modes. This change provides
benefits mostly for hd sequences.

Set up the framework for selectable transform dimension.

Also allowing quantization parameter threshold to control the use
of hybrid transform (This is currently disabled by setting threshold
always above the quantization parameter. Adaptive thresholding can
be built upon this, which will further improve the coding performance.)

The coding performance gains (with respect to the codec that has all
other configuration settings turned on) are

derf:   0.013
yt:     0.086
hd:     0.198
std-hd: 0.501

Change-Id: Ibb4263a61fc74e0b3c345f54d73e8c73552bf926

This commit adds a pick_sb_mode() function which selects the best 32x32
superblock coding mode. Then it selects the best per-MB modes, compares
the two and encodes that in the bitstream.

The bitstream coding is rather simplistic right now. At the SB level,
we code a bit to indicate whether this block uses SB-coding (32x32
prediction) or MB-coding (anything else), and then we follow with the
actual modes. This could and should be modified in the future, but is
omitted from this commit because it will likely involve reorganizing
much more code rather than just adding SB coding, so it's better to let
that be judged on its own merits.

Gains on derf: about even, YT/HD: +0.75%, STD/HD: +1.5%.

Change-Id: Iae313a7cbd8f75b3c66d04a68b991cb096eaaba6

Change-Id: I01dd96da18fe22450bb372b10ba90eab52e29bdf

Resolved the decoder mismatch issue due to quantization parameter
threshold for hybrid transform coding. The macroblock dequantizer
initialization is moved to be performed before coefficient
detokenization, since the (de)tokenization is now dependent on the
macroblock level quantization parameter.

Change-Id: I443da4992ebb70ae4114750b2f1363c0c628580e

Change-Id: I4fceefe76b14bd38a358dc4356708f7981bab339

Using surrounding reconstructed pixels from left and above to select
best matching mv to use as reference motion vector for mv encoding.

Test results:
       AVGPSNR  GLBPSNR VPXSSIM
Derf:  1.107%   1.062%  0.992%
Std-hd:1.209%   1.176%  1.029%

Change-Id: I8f10e09ee6538c05df2fb9f069abcaf1edb3fca6

The forward and inverse hybrid transforms are now performed using
single function modules, where the dimension is sent as argument.

Added an inline function clip8b to clip the reconstruction pixels
into range of 0-255.

Change-Id: Id7d870b3e1aefc092721c80c0af6f641eb5f3747

This allows building on MountainLion as the 10.6 SDK has been
removed from the latest Xcode version (4.4 4F250). Also fix
all warnings for that build.

Change-Id: Ib70bca4a25295f13595f0d10ea9f0229631de5a4