One of the things we struggle with at Flex is the trade off between QA and quick turnaround times on bug fixes and new features, which is a pretty common problem in software development. The more time you spend checking your work, the more releases and bug fixes get delayed.
This tradeoff reared its ugly head today with the release of 4.4.23. Our testing missed three or four fairly serious bugs and our frustration was compounded by the fact that they were all quick fixes that could have been easily corrected if only we'd known about them before release time. We invest more technology and resources into software testing than any other rental software provider, but even our process doesn't catch everything.
Our customers are very demanding and we have a large backlog of new features to work on, so in general our philosophy leans toward quick release times. To minimize how this tradeoff impacts quality assurance, our strategy is to use as much automated regression testing as possible, which enables us to have a high level of test coverage without requiring manual test plans and the time it takes to do manual testing prior to a release. Our continuous integration platform is a pretty good safety net, but every net has holes in it.
In our case, these holes fall into two different areas. The first hole is just those things we didn't think of when assembling our automated tests, meaning weird cases that might pop up in the real world but aren't tested explicitly. Usually when we find a bug that the test automation doesn't catch, we start by adding a regression test that successfully duplicates the bug before fixing it. This is called test driven development. It ensures that our test really works and patches that particular hole in the safety net so we don't have to worry about that bug coming back.
The second hole is a big one, and it's user interface testing. Our existing test automation tests the service layer of Flex, which includes business logic like pricing math, availability and the scan process. We do extensive manual user interface testing, but because this testing is manual and therefore time consuming, it's usually focused on testing new features or bugs that were explicitly worked on during a sprint. A full UI regression test for a system like Flex, if done manually, could take weeks, which we think is an unacceptable delay for our customers.
This would appear to leave us with two options: either push releases without UI regression testing (which is what we're doing now) or develop manual regression test plans and invest the days or weeks it would take to regression test the UI for every release.
Today's experience has reignited our interest in pursuing a middle way: automated UI regression testing. In this kind of testing, our QA team would develop test plans that simulate mouse moves and keystrokes, and validate that the user interface behaves correctly in response. This would validate that UI specific logic works correctly and provide a second layer of test coverage for the core functionality, almost like a backup for our existing test coverage.
We're looking at a tool called Selenium, a popular open source testing tool for situations like ours. There are lots of other tools, but front end test automation is a notoriously difficult thing to set up. You need a specialist (luckily we have one) and it takes time to get the process up and running, but I think it's worth it. Over the weeks and months to come I imagine we'll start with automated test coverage for core functionality: quotes, the warehouse process and expand our test coverage into lesser used areas as the UI regression test suite matures.
There is no process that can catch every bug, but we can always improve and given our current process, I can think of no better way to get some big quality improvements than adding UI test automation. It will be time consuming at first, and require some discipline to get up and running, but in the end it will mean fewer bugs in releases and fewer morning-after-release surprises for customers.