My favorite would be gpustat [1]. This shows the bare minimum amount of information to let's me know that the training has problems/running well

[1] https://github.com/wookayin/gpustat