How do you imagine they would give feedback on something that's neither technical nor illegal? I'm imagining everything from "we found you too arrogant" to "you smelled awful when you came in"...

For arrogant, I'd try to make the feedback as concrete and specific as possible - "sometimes you gave confidently wrong answers. If you're guessing, it's better to tell your interviewer that. Interviewers typically won't hold it against you if you guess and guess rightly, but if you don't acknowledge you're guessing and get things wrong, it raises questions about whether you know what you don't know." or "sometimes it's great to ignore the spec because you have something better in mind, but on an interview it's typically better to demonstrate your creativity and knowledge while still building to the spec - it makes it easier for us to evaluate you" or "when talking about your last company, you said some things that came across as disdainful about your coworkers. It'd be better to highlight your achievements."

All of those are ways being arrogant can manifest, but they're much more actionable than 'you were arrogant' and unsurprisingly get received a lot better.

I wouldn't comment on smell - yes, that's valuable feedback a candidate really ought to hear from someone, but the risk of really angering them is too high for me to feel comfortable with it.

The thing is you still reduced arrogance to technical correctness. However, what I was trying to get at was, what about cases where the technical correctness is just fine? If it's their attitude or hygiene or something else that you don't like, how do you tell them that?

I was trying to get at the same thing you just said, which is that, like you, most people would become uncomfortable providing feedback on at least some of these. Meaning that you would have to turn away these candidates without any concrete feedback. Now how do you imagine they'll react when they realize most people do get feedback but they didn't? Is their reaction (which might result in bad publicity) a risk you and your company really want to take? For what gain?

So the thing is, I think arrogance is typically reflected in actual deficiencies in interview performance. If it isn't - if it's just a vibe that the interviewer got with no concrete implications for how they work with others, solve problems, or communicate - then I worry taking it into account is introducing bias. If I can't think of a concrete implication that the arrogance had, then I don't think I want to take it into account. (You almost always can identify concrete effects, though.)