In the decode side, frame count accumulation is still existing. This
patch removed part of them. More patch will follow. This should speed up
the decoder.

This doesn't change the encoder side since the counts are useful in
some encoder optimizations.

Change-Id: I91a021859f8d35e46618ea9232083e72a06431c8