Get rid of PC_LIBM

It only existed to not include -lm in the .pc for fixed point builds,
but that is still needed since the float API is still enabled and
will use at least lrint.