In a perfect world, you release a product that is bug-free and works exactly as it should and so there is no need for further testing.
However, both product managers and developers know that it’s not as simple as that. They need a way to make sure that there is a process in place that reveals any issues in code in a live production environment.
This is where testing in production comes in.
But it’s also one of the highly debated topics out there with those who say you should always test in production, and those who are more wary of the concept and say you never should.
In this article, we’ll look into these two different perspectives and share our own point of view on this controversial topic and we’ll guide you through the best ways to reap the benefits of this type of testing.
What is testing in production?
To keep it short and simple, testing in production is a software development practice of running different tests on your product when it’s in a live environment in real time.
It allows you to test new code changes on live users rather than a test or staging environment.
This type of testing is not meant to be a replacement for your QA team or eliminating a unit test or integration test. In other words, it is not supposed to replace testing before production but to complement these tests.
To do or not to do: That is the real question
These are big benefits, and they are enough to create consensus among many developers and product managers who say “Yes, always!” to the practice.
But there’s also another group of developers and product managers who say “No, never!” to testing in production.
On the one hand, they admit all of the great benefits that testing in production can deliver. On the other hand, they also believe that the practice carries too many potential downsides and that its benefits just aren’t worth taking on the risks the practice can bring.
Which side are we on?
We believe testing in production is a cornerstone practice for anyone in the software development world. And we believe it is particularly important for Product Managers, as it gives them a powerful method to generate real-world feedback and performance data they need to make sure they are always building a viable pipeline of products.
You can also check out our memes and gifs for test in production.
But even though we are great advocates of this practice, we still want to consider the point of view of those who are “No, never!” when it comes to this type of testing.
Once we acknowledge these issues, we can start to map out some ways to mitigate the practice’s potential downsides and focus on its benefits instead.
What are the big risks of testing in production?
To be blunt: a lot of things can go wrong when you test in production.
- You risk deploying bad code
- You may accidentally leak sensitive data
- It can possibly cause system overload
- You can mess up your tracking and analytics
- You risk releasing a poorly designed product or feature
The list goes on and on. Anything that can go wrong, could go wrong.
Worst of all— if something does go wrong when you are testing in production, your mistake will have real-world consequences. Your product might crash at a critical moment of real-time usage.
You might also end up collecting inaccurate KPIs and creating issues with your business stakeholders.
Worse case scenario: your poorly designed product or feature might result in multiple paying customers leaving your product for a competitor instead.
Those who say “No, never!” to testing in production are correct to consider the practice highly risky, and we understand why they stay away from it.
And yet, while we acknowledge these concerns, when it comes down to it, we believe that this form of testing is an essential aspect of modern software development.
Why should you still test in production?
When done properly, testing in production gives you some great benefits that you just can’t get through any other method.
Collect real-world data and feedback
Testing in production allows you to collect user data in terms of users’ engagement with your new features. This enables the collection of valuable feedback from the customers that matters the most, which in turn would allow you to optimize the user experience based on this feedback.
This will also allow you to brainstorm ideas for features that you may not have considered before.
Uncover bugs
Since you’re testing on live users, you would be able to discover any bugs or issues that you may have otherwise missed in the development stage. Thus, you can ensure your new products and features are stable and capable of handling a high volume of real-world usage.
It is worth noting that there are certain technical issues that will never show up until you put your product or feature in front of real-world users.
Therefore, you can monitor the performance of your releases in real life so that developers can analyze performance and optimize the releases accordingly.
Higher quality releases
Because you’re receiving continuous feedback from your users, developers can improve the products resulting in high quality releases that meet your customers’ needs and expectations.
Additionally, you can verify the scalability of your product or feature through load testing in production.
Support a larger strategy of incremental release
Testing in production helps facilitate an environment of continuous delivery.
This is especially true when you roll out your releases to a certain percentage of users so that they may no longer have to wait long periods of time before they have access to your brand new features.
This way, you can limit the blast radius as with incremental releases, you would not have affected all of your users.
Perhaps, most importantly: you already are testing in production, even if you didn’t know it!
Most of Agile development and product management’s best practices are forms of testing in development. We’re talking about very common practices like:
- CI/CD
- A/B Testing
- Phased Rollouts
- Canary Deployments
- Blue/green deployments
- Usability Testing
- Smoke & Sanity Testing
If you are following any of these practices—and many more like them—then you are already running tests with real-world users in a live production environment.
You are already testing in production, whether you call it that or not, even if you thought you were in the “No, never!” camp this whole time.
Testing in production done right
If testing in development is inevitable these days, then you should spend less time debating its pros and cons, and more time finding the most effective and responsible way to follow the practice.
We believe in this perspective so strongly that we’ve built an entire product suite around helping product developers gain all of the benefits of the practice while minimizing their risks.
Feature flags – a software development practice that allows you to enable or disable functionality without deploying code – are at the core of this new platform.
By wrapping your features in a flag and deploying them into production without making them visible to all users, you can safely perform all of the testing in production that you need.
With feature flags—combined with the rest of AB Tasty— you can:
- Deploy smaller releases that minimize the impact of failure.
- Only test your new features on your most loyal and understanding users.
- Personalize their tests so they know to expect a few hiccups with the release.
- Immediately toggle off underperforming features with a single click.
Read The Many Uses of Feature Flags to Control Your Releases for more use cases and examples.
With feature flags and a little planning, you can dramatically reduce the risk and increase the sophistication of the testing in production you are already performing.
This means more real-world user data, more reliable products & features, and less worry about seeing how your hard work performs outside of the safe confines of development and staging environments.
