Please review the earlier parts in this series:
Part 1: How to cultivate a culture of quality with automated testing
Part 2: How to foster a test-oriented development culture with code checks
Part 3: How to perform robust unit testing and functional testing
Quality assurance (QA) is no longer the phrase that drove developers to recoil in horror and customers to run away to examine their balance sheets. Testing is quickly transforming the way developers work as the number of solutions and approaches increases substantially, especially in recent years. Given the continued promulgation of compelling solutions when it comes to continuous integration (CI) and new software-as-a-service (SaaS) vendors, testing is now a prerequisite rather than an afterthought. Nevertheless, for many of us it remains a new and foreign concept. Luckily, with the right expertise and tools in tow, you too can become an expert in ensuring the quality of your code and user experience.
At DrupalCon New Orleans, Yuriy Gerasimov (Senior Back-End Engineer at Tag1 Consulting) summarized some of the essential concepts that comprise the testing landscape, whether it means constructing a testing infrastructure that can withstand budgetary limitations or empowering a culture of development that considers testing to be the sacrosanct nonnegotiable that it is. In this four-part blog series, we dive deeply into these ideas, and in this fourth and final installment, we spelunk into the depths of two of my personal favorite areas of testing: visual regression testing and performance testing.
Visual regression testing
Front-end testing in general is quickly expanding as an area of interest for developers, especially in the JavaScript landscape. There are now a variety of tools available that allow technologists to test the performance of page loads and decouple JavaScript as a language with its own infrastructure within an existing server-side milieu of testing solutions. For architects and engineers interested in pursuing decoupled Drupal architectures (the subject of my book Decoupled Drupal in Practice and the theme of a yearly conference in New York City), for example, front-end testing can come in particularly handy.
Visual regression testing is another area that is gaining steam as a realm worthy of considerable attention from the development community. In short, visual regression testing involves the verification that no backsliding of quality has occurred when it comes to the aesthetics of your web implementation. As an example, a simple Cascading Style Sheets (CSS) change can sometimes result in all other tests passing as green but a seriously compromised user experience for users who now have to contend with a button that has unintentionally been positioned off the screen.
Snapping and diffing screenshots
The core of visual regression testing is the screenshot. In short, visual regression testing tools automatically snap pictures at a given time of a web page and diff them against either a baseline or another set of images. One common approach to visual regression testing is to feed the testing tool a baseline set of images for every page and compare all subsequent screenshots to that same foundation. By using this approach, developers can acquire more baseline images and additively improve their consistency in discovering issues.
Nonetheless, visual regression testing can also entail comparing environments against one another, such as your production environment with your staging environment. This can be particularly useful for support requests where baseline images may be of limited usefulness. To illustrate this, consider a scenario where you have both a live environment and staging environment that perpetually generate new images that you can then compare. For many development teams, this means a less challenging implementation.
Weaknesses of visual regression testing
Though visual regression testing is highly valuable, it can also come with a range of issues that are worth considering. For instance, performance becomes a more urgent concern in visual regression testing, because screenshots require test runners to traverse every page in a website, load the page in the normal fashion, and take and save screenshots. This process can often be lengthy and sometimes stretch into the range of ten to twenty seconds of latency. If you have thousands of pages, this amounts to an inexcusable delay.
In addition, storage becomes a fundamental consideration for evaluators vetting visual regression testing as a solution. Every screenshot generated by visual regression testing tools need to be stored somewhere, and development teams and customers alike can find themselves quickly exhausting available real estate for their images.
Finally, false positives are a particularly pressing concern in visual regression testing, because a mere change in textual content can cause visual regression tests to yield errors. For this reason, it is important to ensure that content changes are registered before screenshots are taken, as otherwise, such content modifications will be recorded as regressions alongside more serious issues in stylesheets.
Visual regression testing in the real world
In Yuriy’s past experience, visual regression testing has proven to be a highly useful approach that yields substantial dividends for implementation quality. He describes a previous project he was a part of which entailed a migration of a website from a proprietary system to Drupal—but without a single change to the design, as the customer lacked the budget for an aesthetic redesign. Unfortunately, Yuriy and his team found that automated migration was problematic due to the differences in the data structures between the two systems in question.
Thanks to visual regression testing, however, Yuriy and his team were able to discover problems in the migration that were challenging to uncover during the data migration itself. Without the assistance of visual regression testing, as the client later revealed, it would have been impossible to adhere to the ambitious timeline they had set for the project. The development team was able to iterate through two thousand pages within a single week and a single individual dedicated to quality assurance (QA) owing to the successful visual regression tests.
There are multiple tools available for developers to conduct visual regression testing. Two main approaches are setting up a system that will open a particular set of pages and take screenshots and running visual regression testing during functional testing. If you are looking for the crawler approach, Yuriy recommends using either open-source tools like BackstopJS or Wraith or SaaS platforms like Diffy that help you get started much faster. Diffy, for example, is being used by the Drupal 8 core team (e.g. in the Umami distribution). If you are looking to introduce visual regression testing into your functional testing, you can also check out Galen Framework.
Performance testing
Among all types of testing that are available to customers seeking robust web implementations, performance testing is perhaps the most important to sell to a client, because it takes the most time and effort to implement. A website can undergo a full gauntlet of successful tests only for the development team to discover that it cannot handle 30,000 concurrent visitors. While certain tools such as JMeter can be used to approximate the load of five or ten visitors at the same time, most larger projects require a larger number of users, all authenticated into their accounts, and an appropriate budget is absolutely necessary for this type of load testing.
Fortunately, performance testing is among the easiest solutions to sell to customers, because all clients understand easily that their shiny new website cannot go offline the moment it launches due to an influx of simultaneous visitors. But there are unique challenges to performance testing as well: It needs to be run in a production environment for results to be accurate. Though you can run performance tests on a staging environment and approximate based on that load what the production environment can handle, this may lead to unintended results. Teams can also run performance tests when there are fewer users on the production environment, such as in the wee hours of the morning in a time zone when the site’s typical demographic is fast asleep in bed.
Preflight checks
All of these testing approaches should be conducted as a project is being implemented, but there is one final testing regimen that can only take place just before a site launches: preflight checks. Preflight checks tend to be quite slow, but they do not require considerable time to set up and configure. With preflight checks, you can run through all of the URLs in your web property’s sitemap, for example, and verify that they return a 200 HTTP response code rather than a 404.
Moreover, preflight checks can also be helpful in the case of editor errors, which are common in the world of Drupal. For instance, when an editor accidentally uploads a gargantuan image in the order of ten or twenty megabytes, a preflight check can traverse all web pages and return whether they are of a reasonable file size. And if you incorporate your full testing regimen into preflight checks, your automated tests may also encounter other problems in the process. Preflight checks are crucial to consider before a launch deployment, and it also represents icing on the cake for a client who values successful releases, besides demonstrating the professional caliber of your development expertise.
Conclusion
As you can see, automated testing should no longer be an afterthought, not only in the projects you implement for customers, but also as a cultural cornerstone of your development team. Thanks to the multiplicity of innovations now available in areas as diverse as unit testing, functional testing, visual regression testing, performance testing, and preflight checks, you can rest assured that your team will account for any and all issues that may arise. In this blog series, we only scratched the surface of modern testing, but with these solutions in your toolbox, you can rest easier at night knowing you did the best you possibly could for your valued customers and your treasured colleagues.
Special thanks to Yuriy Gerasimov and Michael Meyers for their feedback during the writing process.
Photo by Alina Grubnyak on Unsplash
Please review the earlier parts in this series:
Part 1: How to cultivate a culture of quality with automated testing
Part 2: How to foster a test-oriented development culture with code checks
Part 3: How to perform robust unit testing and functional testing