The Cost of Quality — The Stages of Mobile Software QA
Quality Assurance is an essential part of software engineering and technical product management that absolutely cannot be neglected. A strong QA process and team can provide immense value, but more importantly, ineffective QA can incur immense cost. Let’s look at the stages of QA, and the mounting costs of failures at each level of the software development and test lifecycle. For the purposes of this article, we’ll look at the details of QA as pertains to a software development team operating under Scrum. If you’re not already familiar with the Sprints and Ceremonies of a Scrum team, you might want to go freshen up on those before proceeding.
Test Cycles
In general, there are three pre-release test cycles for a Scrum-based dev team: Dev Testing, In-Sprint, and End-to-End.
Dev Testing
Dev-Testing is the testing the developer does while they are building a new feature or fixing a bug. Once the code is written, the developer runs it, exercises the main uses cases and (hopefully) the most common failure scenarios, and ensures everything works according to the requirements, and that there are no surprises. This is the absolute cheapest place to find and fix a defect in the code. When the developer discovers a bug during development, they can fix it without any other people or teams getting involved. No additional tracking or paperwork is necessary, it’s all just part of the original Story. Robust development testing, and even further, the creation of unit tests or other automated tests, pays off in droves down the line. Keep reading to find out why.
In Sprint
In-Sprint testing is the first time another person outside the development team gets involved with the product. This is the cycle of Progression Testing, where all the new features, changes and scheduled fixes are tested to make sure they work, immediately following development. If a bug is found during in-sprint testing, the story is reopened, and the bug has to go back into development to be fixed. Defects found by the In-Sprint test team incur a dramatically higher cost to fix. More people are involved across multiple teams, the defect needs to be tracked and monitored, and by the time the bug is found, the developer has most likely moved on to something else, and will have to context-switch back to the original Story to fix the bug. Then once the developer thinks they’ve fixed the bug, it has to go back through the In-Sprint test team to be verified. All these things compound the cost-to-fix, driving it much higher than if the developer hadn’t created the bug in the first place, by executing robust dev-testing.
End to End
End to End testing brings yet another team of people into the mix. The end-to-end test team executes a suite of Regression Tests, which verify that all the features that were previously working before a track of new development started are still working after that development concludes. If the developers broke something that was working while they were building something else, the End-to-End test team should find it. But in addition, everything that was Progression Tested (during in-sprint testing) also has to be added to the Regression Test Suite. That means that bugs in new functionality that were missed by the In-Sprint team could be found in End-to-End testing. For the same reasons that In-Sprint defects are more expensive to fix than Dev Test bugs, End to End defects become even more expensive. The End-to-End test team generally tests a Release Candidate — a version of the product that the dev and in-sprint teams think is releasable to users. If a bug is found at this stage, not only does it need to be raised, tracked and monitored, but also, once fixed, new release candidate builds need to be created, and any pre-release checklists have to be re-executed. By the time a track of work is in End-to-End test, the entire development team has moved on to whole new features — development on those features is interrupted while the End-to-End bugs are fixed, and then resumes once they are. Due to the number of teams, people, processes and rework involved, this is one of the most expensive stages to find and fix bugs.
Production
One final stage where bugs can be found, and where they are overwhelmingly most expensive to fix, is in production. When a bug is found in production, that generally means that the bug made its way past all those layers of testing, was released, and then an end-user experienced the problem and reported it. If this bug is critical enough, it incurs extreme cost to fix. Development, In Sprint, End-to-End and any release management layers are now all involved with fixing this bug and releasing a new version (the hotfix), not to mention the teams that are in place to receive and process these types of reports from users. To compound the problem, there might be multiple new work tracks in different stages of development and testing by this point, and all of those will be affected, and possibly delayed, while people get involved in fixing the production problem. To minimize risk, production problems are also generally fixed on top of the version of the software in which they were found, so developers have to revive old versions of the code, find and fix the bug, and rebuild everything. That fix also needs to cascade to any versions of the product that are in development and testing, which might mean fixing that bug multiple times.
In a well-run, properly equipped development and test organization, there should ideally never be a bug released to production that’s so critical as to warrant a hotfix. If you are regularly hot-fixing your products, you have weaknesses in one or more of your defect discovery layers, and finding and fixing that weakness is the most cost-effective thing you can do for your organization.