When you are only using Linux machines, X mostly just works, while correct terminal bahaviour and appearance depend on configuration and TUIs.
I'm not praising X nor am I shitting on TUIs, also I have indeed had more problems with X in general, but not when both host and guest were Linux machines.
I get this too often:
https://robert.penz.name/354/how-to-fix-the-font-for-virt-ma...
The gtk program decided on some font, but didn't have the audacity to request it as a dependency, so you end up with a GUI that prints crap for all text instead.
The problem is "easy" to figure out, the workaround is "easy" and the long term fix is also "easy" but for some reason, it still happens. Now if it would just fall back to whatever default X font there was, I would see text on all strings even with bad size/kerning/look but nopes. Blank squares is where we are at.
So when someone makes a program that just decides to print "Hello" by just printing out those chars on my computer, with my font (or the default, the system, the fallback) it is kind of nice that I at least can see what it tries to tell me.
X is dependent on you using a Operating system supporting X. Ssh can be used across all operating systems, even as an Android app. Sometimes that can be useful.
There are X servers for most major operating systems, including, AFAIU Android. For OSX it's XQuartz (either native or XTools), for Windows, MingX11 and I suspect WSL, as well as various proprietary tools.
I'm not arguing that X11 is more streamlined than remote SSH (or tmux!) in a terminal / console. But it is far more available than you might think.
Android is an interesting use case. It would probably be mostly useless to SSH from an Android if the TUI ecosystem wasn't so prevalent.
You probably also refer to Windows and MacOS, but that just corresponds to another type of difficulty (Xming, Xquartz), it doesn't make it inherently worse/impossible.
In my experience, it depends. X can be a pain, but getting a compatible font and terminal behaviour on the client can be a pain too.