QualityLogic

BACKGROUND

Please describe your company and position.

I am the Policy and Communications Specialist of Whatcom County Government

Describe what your company does in a single sentence.

Whatcom County government provides the services and infrastructure of a local county government

OPPORTUNITY / CHALLENGE

What specific goals or objectives did you hire QualityLogic to accomplish?

• Assessment of our current digital assets with regard to the ADA accessibility requirements
• Provide technical expertise to assist with our compliance with the new ADA Digital Accessibility requirements
• Training to support compliance with those ADA accessibility requirements

SOLUTION

How did you find QualityLogic?

rfp bidding process

Why did you select QualityLogic over others?

• Good value for cost
• Company values aligned

How many teammates from QualityLogic were assigned to this project?

2-5 Employees

Describe the scope of work in detail. Please include a summary of key deliverables.

Audit the County's digital presence for compliance with the new ADA accessibility requirements. Provide training and consultation on those digital accessibility requirements. 

RESULTS & FEEDBACK

What were the measurable outcomes from the project that demonstrate progress or success?

Substantial movement towards compliance with the new standards and wide-spread understanding and integration of accessibility practices throughout the organization. It is now part of every department's workflow and that is thanks to the good work that QualityLogic did. 

Describe their project management. Did they deliver items on time? How did they respond to your needs?

The team consistantly met deadlines for deliverables and was accessible throughout the process for feedback and revisions. 

What was your primary form of communication with QualityLogic?

Virtual Meeting

What did you find most impressive or unique about this company?

Not only did QualityLogic do a great job of articulating the "HOW" of getting in compliance, they also did an amazing job of articulating the "WHY" it mattered - and that's how you get these changes to stick. 

Are there any areas for improvement or something QualityLogic could have done differently?

A few of the trainings could have been scripted a bit better, but they recovered and provided more materials quite smoothly and we are grateful for their expertise. 

BACKGROUND

Introduce your business and what you do there.

I’m a director with Inspirant Group, a consulting company.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were tasked with building a very large and complex application that required extensive testing. We didn’t have enough resources internally to accomplish this, so we decided to hire QualityLogic to help fill this gap.

SOLUTION

What was the scope of their involvement?

We hired QualityLogic to help test the application from all aspects. This involves testing the different types of web browsers that the app can be used with and testing all of its components to come up with a way to break the system so we can improve it. They work in an agile format, so they do individual and regression testing every two weeks.

What is the team composition?

We work with an automation tester, an automation engineer, two system testers, and a project manager from QualityLogic

How did you come to work with QualityLogic?

We found QualityLogic through an online search. We decided to go with QualityLogic for this project based on their responses in our interview.

How much have you invested with them?

We’ve spent somewhere around $100,000.

What is the status of this engagement?

We started working together in August 2021, and the engagement is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

QualityLogic has done a tremendous amount of individual and regression testings, and they’ve successfully identified several defects or bugs within the application that we are actively working to fix.

How did QualityLogic perform from a project management standpoint?

QualityLogic has been great in terms of project management. They’re self-sufficient and incredibly flexible in their ability to maneuver people to accommodate our needs. Their communication has also been fantastic, and they’re very comprehensive in their work. They join all of our meetings and provide detailed updates on their test cases and all of the bugs they’ve identified.

What did you find most impressive about them?

I’m most impressed with QualityLogic’s expertise in pure testing. That, combined with their excellent communication and the breadth of what they can test, really stood out to us.

Are there any areas they could improve?

I don’t think there’s anything they could have done better, and we’re looking forward to doing additional work with them in the future.

Do you have any advice for potential customers?

Make sure to have a good sense of what your needs are before engaging QualityLogic. They have a breadth of expertise in different areas that will help them identify the right people to step on your project if you let them know what you're ultimately going to need.

BACKGROUND

Introduce your business and what you do there.

I’m the VP of engineering at BrightLine, an ad tech company within the OTT space.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were lacking internal QA support, so we hired QualityLogic.

SOLUTION

What was the scope of their involvement?

QualityLogic tests different software development kit (SDK) products for different platforms, including Apple TV, Android, Roku. They test them on different devices as well. We’ll usually have a kickoff session to gain an understanding of the product and what it should do and how things should flow. 

The team then creates test cases and proceeds to do tests on a per-platform basis. Throughout that process, if there are any questions or ambiguities, they’ll stay engaged to clarify them.

What is the team composition?

We typically work with one QA lead, but it could be a combination of two at any given point.

How did you come to work with QualityLogic?

Our previous CTO onboarded them, and I picked up where he left off.

How much have you invested with them?

The cost ranges depending on the level of support we need. Typically, we engage them for at least 20 hours per week, but it can go up to 40 or more hours. We spend $5,000–$10,000 per month with them, and the total cost so far is in the $50,000–$200,000 range.

What is the status of this engagement?

We started working with QualityLogic around October 2018, and our engagement is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

It’s been tremendous. Before working with QualityLogic, we had a single QA teammate internally, and we were experiencing a lot of bugs and complaints from media partners. 

We saw a huge decrease in complaints after we brought QualityLogic onboard, and we saw a big increase in the quality of our releases. It’s been a tremendous impact overall because they’ve helped us decrease bugs, processes, and things of that nature.

How did QualityLogic perform from a project management standpoint?

We discuss what should be tested during the week, and they give us daily updates on any issues they find and report them in Jira. QualityLogic also gives detailed descriptions of how they found the issue and where, along with how to reproduce it. They have a very good process for that.

We have weekly sessions every Monday, and we have a lot of communication over email, which is pretty clear. We also hop on phone calls if we can’t get the understanding through email. 

What did you find most impressive about them?

This is the first QA team I’ve worked with while at the company, but they’ve definitely been better in terms of process and tracking compared to our internal resources.

Are there any areas they could improve?

It’s not necessarily a bad thing, but, in terms of the collaboration between our companies, having more scheduled sessions would help.

Although we had weekly meetings, more frequent sessions throughout the week would be good as well. It can take me longer to get to an email if I’m swamped with work, so having calls and screen shares would make it easier to describe elements of the project.

Do you have any advice for future clients of theirs?

QualityLogic is very in-depth with the products they QA, but we may have an urgent release, where identifying blockers becomes most important. It’s important to understand the product in terms of what’s critical and what isn’t and how that can impact other aspects.

BACKGROUND

Introduce your business and what you do there.

We are a growth assessment provider for school districts, helping to measure areas of opportunity for students that teachers can focus on. We deliver assessments through a hosted solution, and much of our software is web application based. I'm a QA manager.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We hired them to provide more coverage in our operating system and browser compatibility testing. We have multiple development teams who focus on functional and regression testing, but we don't always have time to test operating systems, browsers, and devices.

SOLUTION

What was the scope of their involvement?

QualityLogic runs compatibility tests on a monthly basis to make sure we have no regression in any of the operating systems or browsers we support. As the partnership has gone on, we've added more coverage across our product suite: roles testing, new feature testing, and testing to resolve production issues. They're also extending our accessibility testing to ensure good keyboard navigation and screen reader coverage for student assessments. We hope to expand that as we grow our offerings. We continue to find uses for them, and they plug holes in our coverage.

What is the team composition?

We work with a project leader. She coordinates a team of about three testers, and they allow some flexibility when we need to scale.

How did you come to work with QualityLogic?

I found them via a web search. We identified vendors who could meet our need, and we completed an RFP process. QualityLogic stood out from the beginning, and we were very impressed by their services and abilities.

How much have you invested with them?

We've spent approximately $70,000 on their services.

What is the status of this engagement?

The first official engagement started in January 2019, and they've done testing with us in every month since.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We track the number of test runs they complete, as well as the defects they find. We've improved both sides of the process, and are pleased with the progress.

How did QualityLogic perform from a project management standpoint?

They're very communicative, and their status reports have the perfect amount of information. We have a daily standup with them, and we also use RingCentral to send messages throughout the day. They're very mindful of resources, and they only make changes when we give them the go-ahead. They continue to improve their processes and are very efficient.

What did you find most impressive about them?

This is a unique collaboration for us, but their flexibility, experience, attention to detail, and communication all stand out.

Are there any areas they could improve?

We haven't run into anything. There has been a learning curve on our side, but I can't think of anything they should improve or change.

Do you have any advice for potential customers?

Make sure that you are clear about your own processes. QualityLogic is more than willing to help you as you solidify those processes, and it is useful to maintain an open communication mechanism for feedback.

BACKGROUND

Introduce your business and what you do there.

I’m the director of software systems quality for a transportation company.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We wanted a nearshore vendor as an affordable alternative for testing resources.

SOLUTION

What was the scope of their involvement?

QualityLogic tested our web and mobile applications, specifically the latter’s flow. Tangentially, the team also looked at our automation infrastructure.

What is the team composition?

We worked with a project manager and an account executive.

How did you come to work with QualityLogic?

I had worked with QualityLogic at another company and their mobile testing expertise stood out. We selected the team for this project because they clearly understood our business domain.

How much have you invested with them?

We spent between $50,000–$199,999.

What is the status of this engagement?

We started working with them in March 2018 and the partnership ended in January 2019.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We tracked results via daily touch points to ensure proper oversight and clarification. Our side actively monitored the testing and execution.

How did QualityLogic perform from a project management standpoint?

QualityLogic integrated seamlessly with our business and provided high-quality services. Their attention to detail was notable.

What did you find most impressive about them?

The high-energy team was engaged throughout the partnership. They asked probing questions and proactively communicated any obstacles or challenges.

Are there any areas they could improve?

The partnership ended before we wanted, which limited QualityLogic’s involvement in the automation initiative. We wanted them to take a more active role with that.

Do you have any advice for potential customers?

Constant engagement with QualityLogic keeps the partnership on track. Clients that treat them more like an integrated team than an outsourcing vendor will get better results.

BACKGROUND

Introduce your business and what you do there.

I’m the CTO of Direct Care Innovations. We’re a healthcare technology company offering SaaS-based software to service providers in the Medicaid and Medicare markets.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were in the testing phase for multiple projects and didn’t have enough internal resources to perform quality tests. We wanted to outsource to a vendor who had the skills we lacked.

SOLUTION

What was the scope of their involvement?

QualityLogic test all of our new developments and bug solutions. They conduct tests on our products’ functionalities, performance, and security. Their team uses their own testing tools and show us analytics on the success of our products. The reports show if bugs have been solved and if a system functions correctly. When they find an issue, they screencast with our developers to make the locating and reproducing process simpler.

What is the team composition?

They provide us with five full-time testers and one part-time tester, who also functions as the project lead. Their team is on constant retainer, so there is always a member available to help us.

How did you come to work with QualityLogic?

I searched for vendors online after performing marketing research. One of our system analysts found them, and we preferred their smaller composition compared to the other behemoth-like companies. We also liked their flexibility toward an open-ended arrangement with an undefined scope of work.

How much have you invested with them?

We’ve spent around $150,000, so far.

What is the status of this engagement?

We started in March 2017, and we keep them on retainer for our testing needs.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We’ve been able to increase the number of test cases we get through drastically. Along with quantity increasing, our KPIs show that the thoroughness of the tests is much better than we were doing previously. Our customers have given us positive feedback on the enhanced usability of our system’s overall performance and security.

How did QualityLogic perform from a project management standpoint?

They provide us with detailed documentation about the issues they find and give us quality suggestions for solutions. We receive daily updates from the project lead, which is one of the best parts of working with them. They deliver everything on time and within the budget we set.

What did you find most impressive about them?

Their entire team is professional, knowledgeable, and willing to go the extra mile—we consider them a partner. We rely on them heavily, and they’ve never let us down, so we find the project to be money well-spent.

Are there any areas they could improve?

Although there have been inevitable bugs that went undetected, it’s infrequent, and they give us a reasonable explanation. Our work is complex, and there will always be some issues that slip through—we can’t blame them for that.

BACKGROUND

Introduce your business and what you do there.

We are a large associate development company, with locations all around the world. I work in software quality.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We needed to support a new file format in our machines. There were some new specifications that were being developed at the same time we were developing our machines. We needed QualityLogic to help develop a suite of files that covered the specifications. We also needed help developing the tools to test the files with our machines.

SOLUTION

What was the scope of their involvement?

We had two main goals. We handed QualityLogic the specifications that were in development for the new file format. First goal: QualityLogic needed to develop a set of files, a plot suite, that covered all of the specifications, including the possible variants, the positive and negative clauses, and everything that the specification says. The specifications were very complex with a core module and several extensions. In the end, there were 300 different plots that covered all the specifications. Our proposal was to make sure that our machines were compliant with the specifications. If a machine is going to consume this kind of file format and be considered "compliant with the spec," it needs to accept certain kinds of files and reject other kinds of files. If all the positive cases pass and all the negatives cases fail, we could be sure that we were compliant with the specification.

The second goal was to develop a set of tools so that our team didn’t need to test our machines manually. Every week, there are 300 untested files. QualityLogic developed a set of tools that were automatically able to connect with our machines, send the files, and report the results. The evaluation needed to be formed according to both our quality standards and those of our development team. Every time our development team made a change to the code, they had to make sure that it was compliant with the specifications.

What is the team dynamic?

I had one person that I interfaced with, and I also spoke to a senior engineer. We discussed requirements and technical matters and also budget, scope, etc. I know they had a team of developers and testers behind them, but I only interacted with these two people.

How did you come to work with QualityLogic?

They had worked with my company 15 years ago for another project. We recalled that their work was very good, so we decided to contact them. It was an internal recommendation from a colleague.

How much have you invested with them?

Our initial budget was around $200,000. We expanded the budget a couple of times because we wanted to perfect some tools and add more capabilities with them, to a final of $250,000.

What is the status of this engagement?

We started working with QualityLogic in the spring of 2016; the first project is finished and the second project is ongoing. The first set of standards was completed in seven or eight months.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

The most valuable metric is that we ended up with 300 valid test cases developed by QualityLogic. We also encountered 20-25 issues in our code that we found thanks to this plot suite.

We didn’t intend to qualify the specifications, but QualityLogic found several issues in the specifications themselves. They were in contact with the authors so they could correct them. There was a lot of success in the requirements. That was very valuable because if we had built the code based on wrong specifications, it would have cost us a lot of effort to re-build later on. That was not expected, but QualityLogic reviewed all the specification and asked questions and proposed improvements which was very helpful.

How did QualityLogic perform from a project management standpoint?

We had a meeting every week with a call. In every call, they told us the progress in the development and the new plots. We discussed requirements and priorities. If they needed to choose between different options for a tool, for example, we discussed the best approach. They would inform us on the budget and the cost of doing a certain functionality. They were very transparent. We didn’t have any problem with priorities or time estimations. If they said they were going to do something in a certain time, they made sure they did. We used GoToMeeting for our communication.

What did you find most impressive about them?

Their commitment is impressive. They were very involved in the project. They cared about the success. They understood our problem and wanted to know how we were going to use the product in order to adequate their job. They went above and beyond our expectations. They were good at avoiding failures and giving suggestions. In the beginning, I was the only interface with my company, but in the end, we saw that it was better if they spoke directly to our engineers. Now there’s very frequent communication between their team and ours. Their work is going well, so there will be other opportunities working with us.

Are there any areas they could improve?

No. If I had any suggestions of improvement for them, I told them and they fixed it right away.

BACKGROUND

Introduce your business and what you do there.

I’m the CTO and VP of engineering for a company that makes mobile and SaaS [software-as-a-service] solutions for emergency and clerical communication management. 

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We needed testing for our mobile apps and web app in a way that could be repeated without staffing our own team. We also needed enough physical devices to test everything properly. We’ve worked with other quality companies that focus on test automation. Despite all the automated tests as we build and execute, we still need manual, ad hoc, and exploratory testing. 

SOLUTION

What was the scope of their involvement?

The relationship was already in place when I joined the company. Now, they provide testing services for our applications. As we add more features, they incorporate new tests and have built up a library of those to determine functionality. If new issues are found, they integrate with our defect-tracking system directly. They’re able to enter what type of issue they find, how to reproduce them, and which accounts and users they’re acting on behalf of when they found it. We have other communication channels between them, the development teams, and the support teams in order to refine and triage items and determine what actions we should be taking.

We had a single main point of contact. At the beginning of the year, he started joining us twice a week in our daily standup meetings. He comes to the office and sits in our scrum. That’s working really well. A few other people are involved too.

How did you come to work with QualityLogic?

They’re a local company, and I think a manager of ours at the time ran into them somewhere. I’m not exactly sure how they got connected because that predated me.

How much have you invested with QualityLogic?

We have spent between $150,000 and $250,000 so far.

What is the status of this engagement?

We started working with them in June 2014 and the work is ongoing. 

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or impact of the engagement?

We have a great relationship, and they do excellent work for us. We have urgent timing on some of the releases that we make to enterprise customers. These tasks have time-sensitive events where we’re releasing things for use around the world. On some occasions, QualityLogic has been asked to help us test things really quickly or over the weekend. They’ve always been responsive and worked with us on those urgent requests.

How did QualityLogic perform from a project management standpoint?

We use JIRA for tracking all of the R&D work. They interface directly with JIRA and our Atlassian accounts. We use HipChat for communication about testing and qualification. We also use email and phone to communicate, and our project manager attends our meetings twice a week.

What did you find most impressive about QualityLogic?

They’ve developed a keen understanding of our applications. They know how they work as well as we do, maybe even better than some people on our team because of their broad usage. They take ownership of what they do and are quite proactive.

Are there any areas QualityLogic could improve?

I can’t think of anything. They offer good service. They don’t get into test automation, but that’s not a downside really.