No human had a chimp ancestor. But humans share a fairly recent common ancestor with chimps.
Some human had Neanderthal ancestors, some did not, as you rightly suggest. However, all humans and Neanderthals share a common ancestor that's much more recent than our last common ancestor with chimps.
Ie we only have 'chimp DNA' in the sense that we share a lot of DNA with them, and that we haven't changed too much since our last common ancestor. Exactly the same is true of us and Neanderthals.
