I agree completely. A common mistake is thinking that performance can be tacked-on later. Thinking about performance up-front can inform elegant designs.
Targeted performance improvements tend to make specific operations more efficient. Elegant architectures tend to completely eliminate huge swaths of duplicate and unnecessary operations, and so typically have the opportunity for much greater performance improvements.
That said, I agree with you. Most of the time, the elegant solution is fast or can be made fast.