I suspect somewhere in the Riemann-Zeta function is a reciprocal that only cancels out all the imaginary terms once and only once, thus any number with more than one factor wouldn't cancel out.

I haven't the math skills to find that reciprocal.