When it comes to kickstarting experimentation within an organization, Lukas Vermeer recommends starting small and (keeping it) simple.
Lukas Vermeer took this advice to heart when he dove head-first into the world of AI and machine learning during the early stages of its development, when there was little industry demand. Through consulting for various companies, Lukas discovered his ideal work environment: a scale-up, where he could put his data and machine learning expertise to use.
Enter Booking.com. Lukas joined the Dutch digital travel company during the scale-up phase and went on to lead the experimentation team for eight years, scaling the team from three people to 30 people.
Once the experimentation team at Booking.com had reached maturity, he embarked on a new adventure in 2021 as director of experimentation at Vista. He is building and shaping the experimentation culture and tapping into the potential of their data, to further Vista’s impact as an industry leader in design and marketing solutions for small businesses.
Lukas spoke with AB Tasty’s VP of Marketing Marylin Montoya about the process and culture of experimentation; from the methods to the roles of the teams involved within an organization. Here are some of the key insights from their conversation.
Get strategic about experimentation
Knowing the purpose of your experiment is key. Lukas recommends focusing your efforts on testing big features that can drive real change or impact the company’s bottom line, rather than UI design.
Ask yourself, “What are the biggest questions that are driving your business case at the moment? What are the biggest assumptions that are behind your strategic planning?” he says. Rather than increasing the number of experiments, focus on the correct execution of more significant experiments.
When it comes to building a culture of experimentation within an organization, Lukas suggests using the flywheel method. The first experiment should garner attention by splitting the company’s opinion 50/50, as to whether it will work. This demonstrates that it can be hard to predict the success of experiments, thereby underlining the “unquantifiable value of experimentation.” We need to acknowledge that it is equally valuable to avoid shipping a bad product (that could reduce revenue), as it is to figure out strategically what you should invest in going forward.
Structure your organization for experimentation success
The way your business and teams are structured will impact how seamlessly your experiments are executed. Lukas recommends that the product development team take full ownership of the experiments.
The experimentation team should be facilitating experiments by providing the tools, education and troubleshooting support to the product development team, who can then run their experiments autonomously.
By training product managers in the process of experimentation — such as the different tests and tools available, their strengths and weaknesses, the assumptions they make and when to use them — they can work autonomously to test their ideas and select from a portfolio of experimental methods in order to make a decision.
There is, however, a social aspect to experimentation that should not be ignored. Given the subjective nature of data interpretation and analysis, Lukas highlights the importance of discussing the outcomes and giving feedback on the experimentation process in order to optimize it.
“The whole point of an experiment is to (…) drive a decision, and the decision should be supported by the evidence at hand,” Lukas says. Just as scientists peer-review their papers before publishing, experiments using the scientific method should follow the same guidelines to document the hypothesis, method, results and discussion in the reporting. (An opinion that has been echoed by 1,000 Experiments Club podcast guest Jonny Longden.)
The biggest threat to experimentation culture: leadership or roadmaps?
When people in product development talk about “roadmaps,” they’re not actually roadmaps, Lukas says. It’s more of a linear wishlist of steps that they hope will bring them to the goal. The problem is that there’s rarely alternative routes or redirections should they stray from the original plan.
It’s hard to change direction at the first failed experiment, Lukas explains, due to the “escalation of commitment.” That is, the more time and energy you have invested into something, the more difficult it is to change course.
So, is it time to ditch roadmaps altogether? Lukas advises that roadmaps should simply acknowledge that there is inherent uncertainty. There are many unknowns in product development, and these only become visible once the products are being built and exposed to customers. This is why the build-measure-learn model works, because we take a few steps and then check if we’re heading in the right direction.
Lukas says the goal should not be to “deliver a final product in two months,” rather you should incorporate the uncertainty into the deliverables and word the objective accordingly, for example: to check if customers are responding in the desired way.
What else can you learn from our conversation with Lukas Vermeer?
- When to start experimenting and how to build a culture of experimentation
- The importance of autonomy for experimentation teams
- The three levels of experimentation: method, design, execution
- How to accelerate the experimentation process
About Lukas Vermeer
Lukas Vermeer is an expert in implementing and scaling experimentation with a background in AI and machine learning. Currently, Lukas is the director of experimentation at Vista. Prior to this, he spent over eight years at Booking.com, from data scientist, product manager to director of experimentation. He continues to offer his expert consulting services to companies that are starting to implement experimentation. His most recently co-authored paper, “It Takes a Flywheel to Fly: Kickstarting and Keeping the A/B Testing Momentum,” helps companies get started and accelerate experimentation using the “investment follows value follows investment” flywheel.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
