Ruben de Boer explains what it takes to create a healthy testing environment that paves the way for better experimentation organization-wide
Ruben de Boer is a lead CRO Manager and consultant with over 14 years of experience in data and optimization. At Online Dialogue, Ruben leads the Conversion Managers team, developing team skills and quality as well as setting the team strategy and goals. He spreads his knowledge far both as a teacher with Udemy with over 12,000 students and as a public speaker on topics such as experimentation, change management, CRO and personal growth.
In 2019, Ruben founded his company, Conversion Ideas, where he helps people kick start their career in Conversion Rate Optimization and Experimentation by providing affordable, high-quality online courses and a number of resources.
AB Tasty’s VP Marketing Marylin Montoya spoke with Ruben about exciting trends and evolutions within the world of experimentation, including the various ways AI can impact the optimization of the experimentation process. Ruben also shares ways to involve cross-functional teams to implement a successful culture of experimentation within the organization and why it’s important to steer these teams towards an outcome- rather than an output-driven mindset.
Here are some key takeaways from their conversation.
The goal should always be outcome-driven
Based on his experience, Ruben believes that one of the biggest pitfalls companies face when trying to kick start their experimentation journey is they focus more on outputs rather than outcomes.
“When a company is still very much in an output mindset, meaning we have to deliver an X amount of sprint points per sprint and we have to release so many new features this year, then of course experimentation can be seen as something that slows it down, right? Let’s say as a rule of thumb, 25% of A/B tests or experiments result in a winner and so 75% of what was built will not be released, which means the manager does not get the output goals.”
In this scenario, experimentation becomes an obstacle that slows down these outputs. Whereas, when a company shifts towards an outcome mindset, it makes more sense to run experiments with the goal to create more value for the customer. With an outcome-mindset, teams embrace experimentation with customers at the heart of the process.
When teams are more outcome-oriented, the product is based more on research and experiments instead of a fixed long-term roadmap. According to Ruben, it’s vital that companies adopt such a way of working as it helps create better products and business outcomes, which ultimately helps them maintain their competitive advantage.
Importance of cross-functional teams
Ruben argues that experimentation is maturing in that it’s becoming more embedded within product teams.
He notes there’s a rising trend of different teams working together, which Ruben believes is essential for knowledge sharing when it comes to learning new things about the customer journey and the product itself. For Ruben, this helps create an ideal, healthy experimentation environment for teams to experiment better and get the results they want.
Ideally, there would be experts in experimentation coming in from different teams sharing knowledge, ideas and insights on a regular basis which helps drive inspiration and innovation when it comes to future test ideas.
The recipe behind the success of these experimentation teams varies and depends on the maturity of the experimentation program and the skills of these teams.
This could start with a look into the culture of the organization by sending questionnaires to various teams to understand their work process and how autonomous they are. This analysis would also help teams to understand what their current state of experimentation is like such as how accepting they are of experimentation. This helps to devise a strategy and roadmap to successfully implement a culture of experimentation throughout the whole organization.
This culture scan also helps determine the maturity of an experimentation program.
“Process, data, team, scope, alignment, and company culture: that’s what I generally look at when I assess the maturity of an organization. Is there a CRO specialist throughout the different product teams? How’s decision-making being done by leadership? Is it based on the HIPPO decisions or fully based on experimentation? Then there’s the outcome versus output mindset, the scope and alignment of experimentation as well as the structure of the team- is it just a single CRO specialist or a multidisciplinary team? What does the process look like? Is it just a single CRO process or is it a process embedded in a project team?” Ruben says.
A world of possibilities with AI
With the advent of AI technology, Ruben believes there’s a lot of possibilities with what can be done with it, particularly in the experimentation process.
While he admits it’s still too early to speculate and that there are also the many privacy concerns that come with such technology, he believes AI can bring a lot of exciting things in the future.
“It would be so nice to have an AI go over experiments on the product detail page with all the results and all the learnings, and just ask the AI, what did I actually learn and what would be good follow up experiments on that? And that would be enormously interesting to have an AI run through all the experiments in the database,” Ruben says.
Therefore, Ruben admits there are a number of possibilities of what teams can do when it comes to designing experiments and saving time and steps in the experimentation process.
“And just think about maybe three or four years from now, everyone will just have an AI app on their phone and say, I need to buy this and I will buy it for you. And maybe a website with only AI apps on it to purchase stuff, who knows? And then optimization becomes very different all of a sudden.”
There’s also significant potential with AI when it comes to changing the way people work as well as provide inspiration and ultimately optimize and bring innovation to the experimentation process.
“Maybe based on all the input we give from chat logs, social media channels, reviews, surveys, we can make the AI behave like a user at some point in the future somewhere, which you then don’t have to run user tests anymore because you just let AI see your website.”
What else can you learn from our conversation with Ruben de Boer?
- Evolving trends in experimentation
- His take on change management to help organizations adopt experimentation
- His own experiences with building cross-functional teams
- How to tackle resistance when it comes to experimentation
About Ruben de Boer
With over 14 years of experience as a lead CRO manager and consultant in data and optimization, Ruben is a two-time winner in the Experimentation Elite Awards 2023 and a best-selling instructor on Udemy with over 12,000 students. He is also a public speaker on topics such as experimentation culture, change management, conversion rate optimization, and personal growth. Today, Ruben is the Lead Conversion Manager responsible for leading the Conversion Managers team, developing team skills and quality, setting the team strategy and goals, and business development.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, CMO at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
