Welcome to the first post within our new ‘Feature Experimentation’ series, where we’ll be broaching different topics related to this modern and essential practice in modern product development.
In this series, we’ll be introducing various scenarios where you can reap the benefits of feature experimentation as well as other relevant guides to help you on your experimentation journey.
In this first post, we will list and discuss some essential best practices when it comes to feature experimentation to ensure that your experiments run smoothly and successfully.
Why running experiments should be a central part of your product development process
Running experiments has become a growing, popular trend and a necessity to develop high quality features and products.
Such experiments are key in helping you uncover usage patterns and to give you insight on how your users interact with your products.
Therefore, experiments are a great way, particularly for product managers and product teams, to validate product quality and to ensure that a product aligns with business objectives.
To measure the outcome of your experiments, metrics can be used to help gauge how your customers are reacting to the new feature and whether it meets their expectations.
This means that experiments help you build and optimize your products so you can make sure that you’re releasing products that can guarantee customer satisfaction.
Experiments are also a great way to learn and prioritize resources so that product teams can focus on the most impactful areas for further iteration.
Experiments can come in different forms and these include tests such as A/B testing and multi-armed bandits.
What exactly is feature experimentation?
We talked generally about experiments in the previous section but in this series we will focus on a specific type of experimentation.
As the name suggests, feature experimentation involves feature testing or running experiments on developed or modified features with live users in order to see whether they’re performing as intended.
When we talk about feature experimentation, we’re referring to certain areas within your product that may have issues and need further optimization and improvement.
These features are ones that define the functionality of your software which make the product as a whole more effective and the overall user experience better such as a sign-up flow, a referral program, a purchase funnel or pricing offers, for example.
In other words, features refer to complete parts of your product that often involve multiple stakeholders or teams and are tied to your internal processes or business logic.
These are the features that often have a major impact, positive or negative. As a result, such features need to be tested to avoid the risks associated with blindly launching them into the wild without a clear understanding how they will perform or what their impact will be on revenue and sales or on product usage, for example.
Thus, your team can compare different variations of features with users, instead of going for a full bang release, and see which one confirms your initial hypothesis and shows a positive impact.
This way, only your best features reach your customers after looking at the data that points to the better performing variation.
Experimentation will essentially give you the data you need to do exactly that. Once the winning feature is determined, it can then be rolled out to the rest of your users with the promise of a great user experience.
Some essential best practices for running impactful experiments
As we’ve just seen, feature experimentation and experimentation in general is an indispensable tool for any modern tech and product teams.
In this section, we will discuss some general best practices when it comes to running experiments so you can achieve the best results and avoid any missteps in your experimentation journey.
Create a culture of experimentation
This should go without saying but in order to get started with experimentation, you need to build and nurture a culture of experimentation within your organization.
Some factors will come into play during this process such as your company size, your team’s workflow and capabilities and the type of industry and market you’re operating in.
What this essentially means is that you primarily need to have a clear strategy and roadmap in place so that your teams are aware of the main business objectives to build efficient tests.
We will look into building an experimentation roadmap in another post within our Feature Experimentation series so stay tuned for that!
In the meantime, what is important to note is that this roadmap will serve as the key to link business objectives with product managers’ ideas in order to execute tests and experiments and to be able to set and track the right metrics.
Furthermore, having a culture of experimentation will enable you to make data-driven decisions.
The data gathered from your experiments will allow you to determine and measure the impact of your ideas to see how they resonate with your customers, enabling you to have a clearer understanding of your target audience’s needs.
Building such a culture means you will need to have the right tools in place to help you segment your audience accordingly and tools that will also help you to collect the appropriate metrics and to analyze the results.
Just as important is having and investing in the right people, management and infrastructure to get the most out of experimentation.
However, keep in mind that building this culture of experimentation doesn’t happen overnight.
It requires time and effort but with the right mindset, you can start nurturing this kind of culture within your organization and motivating your team to get started on their roadmaps.
Make it a team effort
To embrace experimentation as part of your company culture, all the relevant teams need to be involved in product or feature testing and not just engineers and developers.
It is important to remember that a good experiment comes as a result of well-defined, shared goals and metrics by all stakeholders.
For example, as mentioned previously, experimentation is a great way for product teams to test out their ideas so everyone needs to be part of the brainstoming process and to look at experiments as a learning experience even if they failed.
In fact, sometimes, it is failed experiments that give the best insight. Any data and learnings gathered from experiments, then, will need to be shared widely among teams so everyone gets a chance to review the results and take the necessary action.
Increasing experiment visibility will allow more people within an organization to clearly see the benefits and processes underlying this practice and highlighting the success and areas of improvement boosts engagement so that they can share their own inputs thereby further instilling a culture of experimentation.
Product managers, in turn, can empower the rest of the teams to be part of the decision-making process on how to improve and optimize products so experimentation becomes a collaborative effort.
It also holds them accountable for the experiments they run so that there is a shared sense of commitment. The earlier a team is involved, the more invested they’ll be in the experiment.
Make it easy
You want to build a culture of experimentation, great, but it’s also important not to make it too complex or a time consuming process that ends up discouraging your team from running their own experiments.
Remember, experimentation should be a collaborative effort, as mentioned previously. Often, experiments may involve cross-functional teams depending on the type and the scope of the experiment you’re looking to launch.
At the same time, there shouldn’t be too much dependence among teams. We already mentioned that every team, and not just development and engineering teams, should be able to run their own experiments.
Feature flags are one way to decrease risk of running experiments by decoupling release from deployment so that all teams feel confident enough to execute experiments. We will go into further detail on that later.
Set realistic experimentation goals
The goal of running experiments is to improve your product for your customers. The results gathered should give you sufficient data to enable you to make informed decisions to optimize your products.
To be able to obtain relevant data, you will need to have a specific goal or objective that will lead you to create a viable hypothesis that you can prove (or disprove).
This is why having a roadmap, as mentioned previously, will be important to allow you to focus your tests so you can get the right data with statistically significant results.
Also, remember that it’s not always possible to test everything. This means you will need to channel your testing energy into running experiments that are relevant to your goals and objectives.
Additionally, some companies may not have a high volume of traffic or users to be able to test everything. This is especially true for feature experiments. A feature needs to receive enough traffic when running A/B tests on this feature in order to generate efficient results.
In sum, good tests or experiments should be focused enough that they give you relevant results and data to improve your products to ultimately ensure customer satisfaction.
Learn from failure
If an experiment goes wrong for any reason and you don’t obtain the results you were expecting, this doesn’t mean that the experiment was a waste of time.
Failures when it comes to experimentation can be considered as a learning experience. This encourages your team to take more risks and boosts creativity.
As a result, implementing experimentation as part of your company culture, regardless whether your experiments turn out to be successful or not, means that it becomes embedded within your team’s natural workflow.
Also, remember knowing what not to do will actually help in improving your product by preventing you from implementing ideas that didn’t perform well so that you know it’s time to move on to the next idea.
Consider the metrics
If you want to make the most out of your experiments by making data-driven decisions then you need to carefully consider the metrics you will track to help you judge whether your feature was a success such as clicks, registrations or sales.
This is an essential best practice as good, efficient experiments are built around a specific goal or metric- the key is to keep a certain focus during experiments, as already mentioned, so as not to deviate from the original goal and lose sight of why you were conducting the experiment in the first place.
This all means that you need to basically tie your experiments to specific KPIs so you can track and analyze the impact of your experiments.
Choosing the right metrics serve as a baseline for your KPIs to enable you to track the results of your experiments so you can make sound decisions.
Target the right audience
This may seem like a no-brainer but to get the results you need to improve your products, you need to choose the right audience to give you those results.
Proper targeting will allow you to see what kind of changes you need to make to your feature variations and consequently, you will be able to tailor the user experience according to the needs of a specific set of users.
This way, product managers can gain valuable insight into their target audience by observing how they interact with different variations of a feature, allowing these managers to validate theories and assumptions about a certain audience.
There are many ways you can go about segmenting your audience, which includes by region, company, device, etc. It will ultimately depend on your own unique objectives.
Remember that to target the right audience, gather the data and analyze the results, you will need to have the appropriate tools at hand depending on your business objectives and teams’ preferences.
Consider the duration of the tests
With feature experimentation, you need to run these experiments for long enough time so you can gather enough data to yield statistically significant results.
Click here to read more about statistical significance and type 1 and type 2 errors which may occur during experiments.
This is important because statistical significance indicates that the results of your experiments can be attributed to a specific cause or trend and are not just a random occurrence.
Therefore, as you start to build your roadmap, you will need to include guidelines for the scheduling and duration of your tests in order to standardize workflows for your team.
However, keep in mind that having a sufficient sample size will be more important than the amount of time an experiment runs.
Use feature flags for safer experiments
For some, the idea of testing in production seems risky and stressful.
However, there is a way to run feature experiments safely without any headaches.
Feature flags are software development tools that decouple deployment from release giving you full control over the release process. In that sense, feature flags can be considered as the foundation of a good experiment.
Feature flags allow you to safely conduct experiments by turning on features for certain users and turning them off for everyone else. If anything goes wrong during your experiment, then you can easily turn off the faulty feature until it’s fixed.
Using feature flags alongside feature experimentation will help you maintain the continuous delivery momentum that is required from modern software development while minimizing the risk of disgruntled customers due to an unstable release.
Furthermore, once you have completed your experiment and obtained the results, you can implement the necessary changes through progressive rollout to further test how these new changes perform with users.
Therefore, through progressive delivery using feature flags, you can introduce changes slowly to your users to ensure a smooth user experience before releasing them to everyone else.
Embrace feature experimentation as part of your company DNA
Some of the biggest companies have achieved their market leadership position precisely because they have embraced experimentation as part of their culture.
Therefore, feature experimentation, when done right, will allow you to make more powerful decisions based on quantifiable data straight from your users.
This means that instead of making decisions on a whim, experimentation will demonstrate what works and what doesn’t based on mathematically-sound data.
Experimentation is one of the most important capabilities offered by many feature management tools.
Our own feature flagging solution, for example, offers an experiment platform that runs A/B tests to track the business impact of feature releases.
This means that everyone has the tools and confidence to take part in experimentation.
For product managers, in particular, it gives them the power to set up, monitor and release confidently without waiting on engineering teams to run the experiments for them through a simple, easy-to-use dashboard.
Our platform focuses specifically on more advanced server-side experiments that allow you to test deeper modifications tied to your back-end architecture using feature flags where you can then measure their impact on the user experience and business.
Find out how AB Tasty can help you transition seamlessly into the world of experimentation by signing up for a free trial.
