Loop has always relied on a recruitment-first approach to hiring QA, labor intensive, but which consistently helps us punch above our weight class. Recently, as an experiment, we posted an QA Automation Specialist position on LinkedIn. The results provided some really interesting insights. The job description included a couple of criteria: candidates needed to be based in the States, willing to work remote, and have a min one year automation experience with Cypress/Playwright. Over the course of a 5 days, we received 1,100 applications! We narrowed down the applications to approximately 200 that met our experience criteria. We then sent these candidates a very brief competency check, designed to test basic automation skills like nested locators and variable usage. It should take no more than 20 minutes even with a year of experience. I do want to acknowledge that I know it's incredibly hard to get a job, and asking candidates to complete a competency check while applying for multiple positions is demanding. On the other hand, our team couldn't possibly interview 200 people. These checks are absolutely necessary to reduce the number of candidates before investing our team's time. Even interviewing 20 candidates is a significant commitment. Of those who responded: -58% dumped the check into GPT and submitted it without ever trying to run the code. There are a couple of dead giveaway patterns. -28% submitted code that didn't pass for other reasons. -Only 14%, or 7 candidates have submitted code that passed so far. Our Takeaway: -The effort needed to identify qualified candidates suggests startups/small teams will clearly struggle with hiring for technical QA roles and may compromise on candidate quality. -Many applicants used GPT to complete the check without running the code, showing a trend of reliance on AI tools that can hide true skills. I love AI, just make sure your code works. -The high failure rate on a simple check highlights a major skills gap in the market for automation engineers. -This experiment confirms the need for pre-interview screenings to efficiently manage the recruitment process and find qualified candidates. Effort likely most teams don't have the capacity for. What's been your experience hiring? #qualityassurance #hiring #qa #softwaretesting
I’ve seen some pretty shocking things interviewing automation testers before, like “not being used to having conditionals” when writing fizzbuzz out *manually*. I find much better candidates apply to “SDET” roles with slightly higher requirements like 2+ years of programming in X language. They can learn playwright fast if they can really program. I think the reason is that the title covers programmers and also those who use no-code/low-code tooling (even things like playwright codegen) so the range of skills is insanely high.
Butch Mayhew
8mo
Wild! Wonder if there is an opportunity to promote those that put in the time and passed the homework (with their permission of course), maybe once the role has been filled.
Pramod Yadav
8mo
I echo with your takeaways. Pre interview screening is indeed one of the best ways to increase chances of finding the right skilled candidates in a limited amount of time.
Matt Franz
8mo
Heard of similar results for DevOps engineers as well.
Was it just an experiment or actual hiring?
Pratik Patel
8mo
Ben F., thank you for sharing your insights! We also recently faced similar challenges while hiring for an actual Python Automation Engineer at Alphabin Technology Consulting. We've received 200+ applications in 2 days, hence implemented a screening test to manage the volume. Our experience was very similar to yours: • 50% of candidates submitted partial solutions, some clearly generated by ChatGPT and other tools • 30% provided promising code that didn’t execute properly, suggesting reliance on AI without thorough testing. • 17% didn’t respond at all to the assignment. • Only 3% submitted well-organized, functional working code that progressed in our process. Therefore, I agree to the fact that pre-screenings are really crucial.
Karin van Blerk
7mo
Hi Ben F., completely understand this experiment, hence we created a custom tool combining the strength of AI and the test intelligence of our experts. It solves the challenges of interviewer bias and the capacity of available senior staff to conduct interviews as well as know when candidates use #AI in the assessments. Read more about it here 😀 https://www.inspiredtesting.com/news-insights/newsroom/607-uncovering-top-talent-with-ai-driven-recruitment-tool
It's heartening to know that an innovative approach is used for hiring, which is capable of picking the candidate who are experts in their craft to some extent.
Daniel Webb
7mo
I pair program with candidates. It’s the best way to assess skill in my opinion.
Helping business leaders with cloud migration, product development and digital transformation leveraging AI capabilities
8mo"The high failure rate on a simple check highlights a major skills gap in the market for automation engineers." This situation exists in market for regular developers/engineers too. Many people chose to take up career thinking it is easy to make money. But they lack the aptitude. We send programming exercise and during the interview ask to make changes to the same code they submitted to find out how well they have understood their own code.