RD search and trellis encoding are still sub-optimal.

Change-Id: I233979909118241a0c78761c1d5c2cd6857915e0