How does Amazon test our mobile apps? Back in May, I was brave enough to chaperone a 10-day school trip for a bunch of 14-yr olds (including my own). One day, after a few hours at a museum in DC, I saw 9 of our kids sitting at the meeting point, intently staring at their phones. Every single one was on their phone - unsurprisingly. As for me, in the last fifteen years, I've been a distributed systems guy who primarily thinks about scale, load and performance for very large systems. Nothing could be more different from the mobile domain. I’ve spent most of my career thinking about how tens of thousands of instances of Service X behave, how hundreds of services interact with each other as throughput increases, and where massive distributed systems like the Amazon Store could bottleneck or fail. But since I came back to Amazon 4 months ago, I’ve been more and more interested in the space of mobile apps and in particular how we test our mobile apps. I've been exercising the Amazon Leadership Principle of Learn and Be Curious to educate myself on an entire domain I knew very little about a few months ago. More and more of our customers are accessing Amazon properties via mobile - and I expect that trend to continue. And Amazon ships dozens of mobile apps - Shopping, Prime Video, Music, Alexa, Kindle, just to mention a few. I've been exploring how we can partner between orgs to improve our mobile test infrastructure for the entire company. For e2e tests, what devices do we run our tests against? Do we test against physical devices, or against virtual devices (emulators and simulators)? When fidelity is critical to us, we run our tests against physical devices. But that comes with limits in terms of scalability, reliability, latency and cost. Virtual devices allow us to scale, increase our test coverage, and shift a lot of our testing left with a small potential tradeoff in fidelity. My belief is that it is not an “or”, it’s an “and”. We should be using virtual devices earlier in the SDLC (e.g. pre-submit), and physical devices later in the SDLC (e.g. post-submit). The criticality of the test also matters: p0s should always run against physical devices. And some tests can only run on a physical device. When it comes to physical devices, we leverage AWS Device Farm heavily. This allows a lot of our teams to focus on their business needs and let AWS solve the undifferentiated work of device provisioning and lab management so that we don't have to. AWS Device Farm is a game changer for mobile testing. In some cases, we need direct access to hardware (or we’re testing prototypes) so we need to have our own private labs also. Last week as I was visiting our San Francisco offices, engineers from Amazon Music were kind enough to show me one of the many labs around Amazon. This is the part of it that didn't have any prototypes or confidential devices 😀
Satyajit Malugu
3mo
For scale and scope of Amazon , things could be different but my 10+ years of mobile testing experience so far finds no advantages running automated tests on real devices. Modern simulators and emulators are very fine for 98% of testing and for the rest of 2% we can rely on manual device testing. Our manual regression happens weekly and we do that on real devices anyways. Real device testing is an extra cost and lower reliability and scalability.
Rustem Kamun
3mo
I understand that fidelity is about accuracy. But in this situation, what do you mean by fidelity or high fidelity?
Aaron Rehaag
3mo
Too bad you didn’t get to see the Google device labs. Some of them blow me away with their sophistication.
Chris Traganos
3mo
that is one clean test lab, I'm impressed!
Mohit Sehgal
3mo
that brings back memories, just seems scaled down, do we still have a secretive room to the left of it - OGs - Moon Seop Kim Deepak Srikanth Sripal Vurukonda Hari Nagendra Veeramahanthi Karthiek Satyagari
Bruno Guardia
3mo
Look at Appwright, just announced, working in the same problem area https://www.empirical.run/blog/appwright
Jason McMahon
3mo
This brings back memories hah. I was the dev manager for Amazon’s first Mobile App team. We started building a BlackBerry native app first (they were all the rage), then quickly pivoted once Jobs announced the iPhone. This used to be a drawer in my rolling cabinet, with an increasingly fragmented collection of devices and charging cords. This looks much better organized 😃
The fact that Amazon has private labs for prototype testing is both fascinating and makes total sense. It’s insane to think about how many apps Amazon ships and how everything needs to run seamlessly on millions of mobile devices. Your team's approach seems like a good mix of scalability and precision. Keep up the hustle, man. This is next-level work!
Ed M.
3mo
I remember when I was in charge of music certification for tablets - you needed physical devices due to streaming app requirements of running for a certain number of hours (w logs). It was really annoying to test long running tests that failed towards the end. No easy way to parallelize testing.
thanks for the hospitality, Hari Nagendra Veeramahanthi and team!