Fix the corner case and use the right rate cost udpate for rd_debug.
This would make the var-tx pass rd_debug test.

Change-Id: Ib0fbd2d73030c0d150222c6b7c2dfffc0c6af085