While lean agile software development practices enable rapid feature development, shorter, fast-paced iteration cycles don’t mean quality should fall by the wayside. In the ’80s, borrowing from the manufacturing industry and fueled by the Deming movement, new quality assurance (QA) initiatives introduced monitoring of software engineering processes. From the start, QA involved testing for quality and took place in its own organizational department. (You can read a brief history of QA here.)
Fast forward to 2018, where organizations strive to address customer demand for more sophisticated features and the need for rapid release cycles. Smart organizations address these challenges head on by implementing lean agile software development practices. In an agile approach, requirements and solutions evolve through the collaboration of cross-functional teams. In large part, however, organizations have not evolved their traditional QA practice.
Agile’s core tenets – adaptive planning, evolutionary development, continual improvement, and rapid and flexible response to change – do not play well in this siloed QA world. The result? Six QA testing flaws that impact the scope, speed, and quality of software releases.
QA Testing Flaws: Scope, Speed, Quality Tradeoff
Developers write code, then pass that code to testers, whose job it is to discover defects. Testers document the bugs they find and pass those findings back to the developers, who have moved on to other assignments and now must context switch. This process continues until the priority and severity levels of the identified bugs are considered to be at an acceptable level, at which point the release is cut and the product ships.
Scope, speed, quality – something’s got to give (or so the saying goes). Scope is arguably easiest to cut by limiting the features in a release. When scope and quality are essential, release dates are pushed out (sometimes repeatedly). Quality, in turn, often suffers when scope is locked in and speed is of the essence. In these cases, some bugs don’t get fixed, and new bugs may be introduced due to the rush to meet prescribed release dates. Developers may also address the easiest bug fixes first, leaving major issues hanging.
Leadership teams often debate the scope, speed, quality tradeoff, but I’d argue that the real question has to do with why the tradeoff exists in the first place. Do developers and QA testers really need to hand code back and forth, one group focused exclusively on coding, and the other focused exclusively on identifying bugs?
QA Bottlenecks
As we’ve seen over the last decade, organizations are keeping up with customer expectations and demand by adopting lean principles and DevOps practices such as CI/CD (continuous integration/continuous delivery). While development and operations have evolved substantially, it’s not uncommon for QA organizations to remain unchanged, much like they did in the ‘80s. Siloed QA organization structures result in a disconnect that impedes rapid development and rapid delivery. While QA testing improves code, QA doesn’t address the scope, speed, quality tradeoff dilemma.
Organizations can benefit greatly from a DevOps transformation and a QA testing reboot that eliminates a number of development/QA handoffs and streamlines QA.
QA Safety Net
Let’s talk for a minute about QA testers whose job it is to find bugs. What they’re really doing is providing developers with a safety net. Developers know that they have another group who will perform a full battery of tests and report back any changes that are needed. Continual cycles of siloed handoffs between development to QA continues until both parties are satisfied or time is up. What was intended as a final check system has become the first line of defense.
This model is deemed necessary because the conventional wisdom is that developers aren’t capable of writing code correctly the first time (at least on a consistent basis). As a result, the QA tester’s cleanup efforts have become necessary at every stage of the software development life cycle (SDLC). But what if QA instead focused on higher-order tasks that expedite delivery?
Shift Left Testing
These days, there’s lots of buzz around a “shift left” testing approach. In this approach, software testing is shifted as early in the SDLC as possible. (You can read more about scrum teams here.) To that end, many organizations integrate one or more testing resources into the scrum team and then attempt to automate testing during the sprint to try to limit the need for scope, speed, and quality tradeoffs.
While this approach can help address some of the problems with siloed QA, it really just shifts bottlenecks left to an earlier stage of the process. Oftentimes, the QA automation resource has trouble keeping up with multiple developers on the scrum team, and the developers end up spending more upfront time fixing code than actually coding new features. Teams still face the age-old problem of not having enough automation engineers to keep up with the development team output.
In the end, this style of shift left testing doesn’t compensate for the absence of a solid, truly integrated DevOps pipeline, and it doesn’t make definitive quality improvements or expedite releases.
One may then ask this: how do you achieve the illusive shift left methodology? First, every code push to the repository must have an accompanying testing change (unit, functional, security, performance). This principal becomes part of the “definition of done” for a story. In other words, the story is not complete until both coding and testing are completed within the sprint.
Second, QA is often a shared service outside the delivery teams, where the QA resource on the delivery team is not a full-time, dedicated resource but instead supports multiple teams. Finally, the truth is that you can’t achieve continuous, unattended testing without a solid, truly integrated DevOps pipeline in place.
Continuous Unattended Automation
DevOps principles such as CI/CD can significantly speed up the development process. However, let’s say you’re manually testing 3,000 scripts in your organization. Whether you realize it or not, you’ve set yourself up for another series of bottlenecks in your transformation journey.
Here’s why. Enterprise organizations often single source their development staff to one vendor, and to keep that vendor honest they outsource testing to a separate vendor. The problem? There is no incentive for outsourced testers to automate the manual tests. For one thing, enterprises haven’t had visibility into the bottleneck caused by manual QA efforts. For another, outsourced vendors can make more profit by employing a large number of manual testers than they would by automating the test scripts once and execute the scripts as often as necessary.
These realities aside, let’s say that an organization recognizes traditional QA testing flaws and is prepared to automate testing. Now the organization has a new set of challenges. For example, those 3,000 automated test scripts don’t necessarily cover every possible user scenario. And those test scripts are often shrouded in mystery, as few members of the team usually know how to write them or run the automated test suite.
While automation is critically important for testing and quality, it only tells you if your code is functionally correct for the scenarios you think to code for. It doesn’t tell you about usability, robustness, security, or other aspects of quality that are more difficult to define. Developers also don’t always know good testing automation techniques or frameworks, and in many cases they’re not asking the right questions. (To be fair, this hasn’t been part of their education or job description for the last few decades.)
Automation is a giant leap in the right direction. But it isn’t perfect.
Upskilling
Another popular trend is “upskilling” (also known as “reskilling”), which involves staff transitioning from traditional manual testing responsibilities into new tech engineering roles that are in high demand. The writing’s on the wall, and these days it’s hard to make a strong case for manual testing solutions. More frequent adoption of shift left approaches and automation is reinforcing the need for built-in quality. Vendors want to win recompetes and new work while retaining their existing staff, so they promise enterprise organizations that they’ll upskill their non-tech QA talent. Their goal is to make coding so easy that manual testers can do it.
The problem? QA resources aren’t developers, and they often lack the requisite programming background and technical skills to do the job of engineers. In my experience, only about 5% of manual testers succeed as cross-functional scrum team members (essentially full-stack engineers). They’re simply not set up for success.
What’s Next?
Enterprise organizations need to automate as much testing (and all other manual tasks) as possible so added features flow through the pipeline from development all the way to production without substantial (and ultimately no) lag times.
Shift left testing methodologies, automated QA testing, and upskilling are a good start. All three trends offer vast improvements to dated manual processes, and they’re definitely pointing in the right direction in the journey toward QA transformation. However, solving the fundamental problem with QA testing flaws will require enterprises to challenge assumptions.
For starters, QA can no longer work the way it did in the past (nor should it). The time has come for a quality engineering transformation, which is the subject of my next post.
Also — You can read more about why enterprise quality is broken (and why many enterprises have no idea).
