We also did some amount of training on our own GPU machines on premises.
But we recently started to work with our YC batchmates of https://snark.ai , which enables us to speed up training by A LOT