The “jitter” of PSX graphics is caused by a number of factors: https://retrocomputing.stackexchange.com/a/5021

Incidentally, the Nintendo 64 also used fixed point numbers in RDP graphics instructions, but did not exhibit the same visual artifacts as the PlayStation.