Elon Musk personal brand is incredibly valuable, justified or not.

I don't think that he really runs the company, and kicking him out will most likely not change anything technical, but it will affect the company image among shareholders, most likely negatively.