Yes! I think this is a good explanation. I find it hard to relate two different notes to each other when they are sufficiently different. If they are very different then it obvious but if they are 1-2 tones apart I only really hear them as two different notes rather than one note clearly being higher than the other.
It can get even weirder when you jump a gap more than half an octave. For instance say you jump down 6 notes, absolutely speaking it's lower... but relative to the original "note" (not pitch) it's closer to being above than bellow, which can make it feel higher at the same time.
I can perceive this higher+lower aspect simultaneously in a lot of music and i suspect this is common even.
