In my previous post, I briefly described how leading companies use experimentation to optimize their products and services and evolve them to the point of feeling elegant, efficient, and magical. These companies have developed mature experimentation programs (ExPrs), including the infrastructure, people, and processes for using experiments in areas like product development and marketing.

But most companies are in earlier stages of adopting experimentation. Employees working at these companies may not need to be convinced of the value of running experiments, but they typically lack a roadmap for how to start integrating experimentation to improve their operations. The goal of this series is to provide that roadmap.

In this post, I’d like to define what an experimentation program is and discuss which stakeholder groups should participate in the ExPr for it to be successful.

What is an Experimentation Program?

An experimentation program is the mechanism by which a company uses randomized controlled experiments to generate positive business results. An ExPr is composed of the people, processes, and infrastructure for running experiments at a company.

Technical readers might be familiar with some of the technology required for running experiments, including software for splitting users/visitors/etc (i.e. experimental units) into treatment groups and for computing the results of completed experiments.

But ExPrs are composed of more than just the technology required to run experiments.

An experimentation program can succeed only when the right people are involved. ExPrs require the participation of a cross-functional set of stakeholders – individuals and teams from various business units, data science, and technology. These people are responsible for driving experimentation forward by ideating, planning, implementing, and analyzing experiments.

Exactly how this ideation, planning, implementation, and analysis is completed is the process component of an experimentation program. Processes describe how stakeholders collaborate with one another and with technology to generate positive business outcomes through experimentation. These processes describe how experiment ideas are generated, collected, prioritized, and implemented. They also describe how experimental results are evaluated and what change management is required to operationalize the results of successful experiments.

Subsequent posts in this series will focus on the processes and technology components of ExPrs. In the remainder of this post we’ll discuss people.

Stakeholder Groups

What stakeholder groups should be involved in building an experimentation program? I categorize these groups by their level of direct involvement in an ExPr. Primary stakeholders are directly involved in the planning and execution of experiments. Secondary stakeholders are not directly involved, but must remain informed in order to maximize the probability of adoption of experimental results.

One of the most challenging yet exciting parts of driving an effective experimentation program is that it requires cross-functional participation at multiple levels of the organization. By cross-functional I mean that the effort requires participation from individuals and teams located in various business units (specifically those business unit(s) seeking to optimize their results), data science, and technology.

Let’s briefly describe the roles of these cross-functional groups:

1. Business Units

Business stakeholders are the people who own the process/product/outcome they’re seeking to optimize through experimentation. For instance, a product team may wish to increase the average revenue per sale for a particular product.

The business stakeholders can be thought of as the "customers" or "users" of an experimentation program. They’re investing in the ExPr to improve an outcome measured by a metric that their business area cares about.

Business stakeholders may or may not possess the experience or expertise to understand how experimentation can be used to drive the results they care about. However, they are the subject matter experts of their functional area, products, processes, users, etc. Business stakeholders without this experience may need to be persuaded of the benefits of experimentation. A common cause of concern is when an experimental variant might cause value to be destroyed. In this case, it’s up to data science to present the relevant trade-offs by discussing guardrail metrics, sample size splits, etc.

However, business stakeholders who do possess experimentation experience, or at least who believe in the power of experimentation to drive business change, can be very effective partners when bootstrapping an ExPr.

2. Engineering

Effective experimentation relies on trustworthy and reliable data. This data is typically generated when a user interacts with some part of the organization’s technology stack. A "user" could be internal, such as an employee of the business, or external, such as a paid customer or visitor to a website. Similarly, the org’s tech stack could be the proprietary software it has built in-house, or it could be 3rd party software the company uses (for example, data from an email marketing tool or log data exported from the phone call vendor system used by the sales team), or some combination of both.

Engineering must be involved in the ExPr because this team best understands the systems that will be involved in the actual experiments. For example, when running experiments that involve parts of the tech stack that haven’t previously been involved in experiments, engineers are needed to ensure that the technology is instrumented with the proper telemetry and that these logs can be accessed so experiment results can be analyzed. In more mature experimentation programs, engineering teams might work on integrating their systems with their company’s centralized experimentation platform (we’ll talk more about this in future posts).

In each of these cases engineering management needs to have a seat at the table to ensure that the required efforts are prioritized on their roadmaps.

3. Data Science

Data science plays two important roles in an effective experimentation program.

The first role is tactical. Data scientists are the statistical domain experts. Their background in experimental design and analysis is used to design and rigorously evaluate the results of individual experiments. This expertise helps organizations guard against common statistical issues such as Simpson’s paradox, under-powered experiments, and peeking. In this role, data scientists help determine the parameters of individual experiments such as sample size requirements (How long do we need to run this experiment?) and experiment populations (How do we define who should be included in this experiment?).

The second role is strategic. The data science team should be responsible for driving the experimentation program forward by influencing the cross-functional group of stakeholders (in a collaborative manner). The main idea behind this is that data science is closer to the business than engineering, and closer to engineering than the business. The data science team should act as a bridge between these two stakeholder groups, helping to translate business goals into technical engineering requirements and then helping to validate that what engineering builds meets the requirements necessary to run effective experiments.

This second role should be played by data science managers or product managers who sit on a data science team.

Secondary Stakeholders

An experimentation program may also impact a set of secondary stakeholders – individuals or teams that are in some way(s) impacted by the planned experiments, but are not actively participating in driving experiments forward.

One example of secondary stakeholders are business units that share accountability for the business outcome we hope to improve through experimentation.

For instance, suppose the Sales function in a B2C company wants to reduce their costs. They partner with the data science team and decide to test certain automated, low-touch sales strategies for subsets of their prospects. But suppose they share accountability for a metric such as the number of qualified prospects in the funnel with the Marketing function. In this case, the Marketing team is a secondary stakeholder in the ExPr. The marketing team should be made aware of any planned experiments that impact the KPIs for which they’re jointly accountable, as well as the results of experiments that have concluded.

Secondary stakeholders need to be actively informed about the decisions and results of an ExPr. Failure to inform secondary stakeholders can lead to experimental results not being fully embraced by the organization.

Conclusion

In summary, an experimentation program is the mechanism by which a company uses randomized controlled experiments to generate positive business results. It’s composed of the people, processes, and infrastructure for running experiments at a company.

The people involved in an ExPr are composed of a cross-functional group of stakeholders who operate at multiple levels within an organization. These stakeholders can be divided into two groups.

Primary stakeholders are composed of people from individual business units, engineering, and data science. These individuals collaborate to plan, implement, analyze, and (hopefully) operationalize experimental results.

Secondary stakeholders must be informed of experimental progress, but are not actively involved in running experiments. Keeping secondary stakeholders informed is important for ensuring that experimental results are accepted and embraced across the business.

In the next post, we will discuss an interaction model describing how these stakeholders can interact to drive experimentation forward. If you’d like to be notified when I publish this series, sign up below and I’ll email you each post as I publish them.

Opinions expressed here are my own and do not express the views or opinions of my employers.