My answer to that question is: "I've tried management and whilst I was told that I was good at it, I didn't enjoy it much - I discovered that I simply enjoy building and creating things, and that's what I do best".

Nevertheless, I appreciate that it's an irritating question, but I usually put it down to lack of empathy and/or life experience of the questioner. It would seem that some people just can't imagine that other people have lives, thoughts and aspirations that are different to their own.