I generally use rxvt because xterm is too slow. It doesn't have menus or URL highlighting and scrollbars are nonfunctional in the presence of screen, tmux, Emacs or generally any interesting terminal app.
On Windows I use mintty and I turn off scrollbars there. I simply don't use the mouse to interact with the terminal other than to select text, and that's with selection buffer to copy.
Speed is highly relevant to me. Most modern terminal emulators are very slow, most noticeable when you get a lot of output in a panel in something like tmux.
(Simplistic benchmarks that test full screen scrolling usually hand the crown to terminals that don't bother to refresh the screen with everything output, but that's not the only bit of a terminal emulator that can be slow.)
Interesting. For me, there isn't enough integration between the GUI and terminal ... they shouldn't seem separate, but should be bridged to create a coherent experience.
As a suckless terminal user I would say it is not the goal of the terminal emulator to provide scrollback, gnu screen or tmux are far better tools for that purpose.
In the case of tmux, it's only a single line change to activate mouse scrolling. It's nice to have the times when I switch over from a browser and instinctively try to scroll around.
It might be super fast, but I've not really been scroll speed limited.