Are you sure that it wasn't due to the fact that Intel Atom doesn't have AES instruction set[1]? I think openssl uses it a lot so it would explain this huge difference (since looking at the clocks, AVX/SSEs etc. I would expect at most ~50% difference looking at single core of Xeon vs single core of Atom) unless generations of Xeon and Atom were many years apart. I think newer RbPis CPUs have AES instructions so they could be faster than Atoms in applications that utilise it.

[1] https://en.wikipedia.org/wiki/AES_instruction_set