Unbreak non-x86_64

The rust compiler DCE works in a different way so conditionals on `cfg!`
should not be used if the code in the branches may not compile.

While at it move the `use` statements where they are needed to avoid
warnings.

Fix #657