We have a lab environment with multiple phones plugged into a single controller. The controller runs a set of Jenkins agents which advertises the different devices as targets for Jenkins jobs.
Plus, lots of tweaking and perf related fixes to the phone. If you're looking for an off-the-shelf solution, check out Headspin or the Google Cloud offerings.
Thanks for the feedback. I'll look into headspin. I'm not into mobile performance or testing so not sure how I would automate and log the performance on different devices. Honestly I'm not sure what tools I would use on the device to run the tests.
We run functional UI tests with a performance tracer wired up- we already had them setup by feature teams so it was easy to get started. Instead of a Zookeeper scheduler, it’s just Jenkins.
Right now we’ve been able to catch 2-3% relative regressions in things like startup, opening a Google Maps view etc which is pretty good. It was easy to setup and then hard to optimize the variance down so we could get a strong signal.
We’re hiring if any of this is interesting, it’s Golang/San Francisco/Full-Time.
