800k parameters, 600 MFLOPS, with a receptive field of 3 feature vectors

Starting with compute_linear()

Saves ~270 kB of weights in the decoder

Removing some indirections

Only code non-degenerate symbols, which makes the encoder faster

Including removing useless code

Removing one of the 2d conv layers for pitch estimation reduces
complexity without noticeable degradation. FARGAN model has more
adversarial training.
Also, no need for the double precision in the low-pass filter.

Gets us about 2x speedup on x86