In my previous post, I defined an experimentation program (ExPr) as the mechanism by which a company uses randomized controlled experiments to generate positive business results. An ExPr is composed of the people, processes, and infrastructure for running experiments at a company.
The people involved in an ExPr are necessarily cross-functional and operate at multiple levels within an organization. They can be divided into two groups. Primary stakeholders are the individuals from business units, engineering, and data science who collaborate to plan, implement, and analyze experiments. Secondary stakeholders aren’t actively involved in running experiments, but are directly impacted by the ExPr and should be informed of experiments that impact any of their business areas.
In this post, I’ll describe how these stakeholder groups can drive the experimentation program forward through the formation of a working group. First I’ll describe the goal of the working group and what role data science plays in the group. Then we’ll discuss how the working group can drive experimentation forward through a results-focused recurring meeting. Finally, I’ll provide tactical advice for improving the working group’s probability of success.
The Working Group
So we know what an experimentation program is and who should be involved. But how do we actually drive the ExPr forward?
The experimentation working group is a group of individuals whose goal is to implement the experimentation program i.e. to drive positive business results by planning, running, and analyzing experiments.
The working group (WG) is composed of key decision makers from each of the primary stakeholder groups that:
- Represent their stakeholder group, in part by sharing relevant information from that group.
- Are accountable to the working group for the completion of tasks assigned to their stakeholder group.
- May be responsible for completing certain tasks, but may delegate tasks to individuals on their teams.
Data Science Participation in the Working Group
Which data science team members should participate in the working group?
The data science team plays a strategic and tactical role in the experimentation program.
Tactically, data scientists are the statistical domain experts. The working group should include a senior data science manager who can be the group’s stats expert and can assign specific tasks to individual data scientists on the data science team. This manager might to include the data scientist performing the work in the working group. Participating in the WG discussions will provide that data scientist helpful context for designing a proper experiment.
Strategically, the data science team is responsible for driving the experimentation program forward. This requires someone to play a program manager role. This person is responsible for organizing the working group, leading the working group meetings (we’ll discuss this next), and influencing the working group participants. From experience, I believe this role is best played by a product manager reporting in to the data science team. Note that over time it might be necessary to evolve the experimentation program manager role into a full-time position.
The Working Group Meeting
The working group should meet regularly to achieve it’s goal of implementing the ExPr. During these meetings potential experiments are discussed, action items are defined and assigned, and progress against these action items is shared.
Specifically, working group participants should collaborate to:
- Discuss hypotheses that the business wishes to test
- Prioritize the set of proposed hypotheses/experiments
- Determine the action items needed to implement prioritized experiments
- Determine who is accountable for those action items
- Determine timelines for those action items
- Communicate status updates
- Discuss the results of ongoing/completed experiments
Don’t use the WG meeting to flesh out the details of each experiment. That work should be performed by the WG participants and their teams in-between meetings_.
For example, once a hypothesis/experiment has been prioritized, data scientists will design the experiment and determine parameters such as the required sample size outside of the meeting. The data science representative will then share relevant details during subsequent WG meetings.
Completing certain tasks will require multiple stakeholders to collaborate. In such cases these stakeholders should hold additional meetings (see Meeting between Meetings) to complete these tasks.
Tactical Tips for Running an Experimentation Working Group
Lets discuss a few tactical tips for improving the the working group’s probability of success. This advice is borne from my experience leading and participating in multiple WGs tasked with driving experimentation in multiple business/product areas.
Bootstrapping a Working Group
Think about your most productive working relationship. How does it feel to work with one another? What are the qualities that make this relationship so productive? And perhaps even enjoyable?
You and your coworker might breeze through design discussions, effortlessly parallelize your efforts, and make great recommendations to one another. Even when one of you criticizes the other’s work, you both know and feel that the criticism comes from a place of positive intent because you’re both out to achieve the same goals.
Chances are this relationship wasn’t this productive when you first began collaborating. Perhaps there was a "feeling out" period that allowed you to build trust in each other (this is especially true if you have very high standards for your work). Even if you gelled early on, chances are your ability to work together has still improved over time.
Similarly teams working together for the first time are rarely as productive as teams that have a track record of achieving goals together. This is important to keep in mind when bootstrapping your experimentation working group.
It’s unlikely that the stakeholders participating in the ExPr will work together productively or efficiently at first since these different teams may never have collaborated previously. That’s ok, Rome wasn’t built in a day.
Your first order of business to articulate and align on the experimentation program’s goals. Instead of jumping directly into hypotheses and experiments, use the first few meetings to discuss why experimentation is important for your company and what business value you expect the group to deliver. Explicitly state each stakeholder’s role in this mission.
Aligning on the WG’s goals provides the group a shared sense of mission and vision. When coupled with trust that will be developed as individuals hold each other accountable for specific tasks, this initial alignment will transform the group into a highly productive, efficient, and motivated unit of experimentation practitioners.
Meeting Cadence
The WG should meet often in the initial stages of the ExPr in order to establish a tight feedback loop. Meeting frequently helps the group learn how to work together and establish a productive cadence. I’ve found that meeting biweekly at this stage strikes a balance between allowing a productive cadence to be established and providing stakeholders adequate time to make progress on their individual tasks.
As the ExPr matures the meeting cadence can be extended to monthly. I caution against doing this too early. Cancel an individual meeting if necessary, but keep the biweekly event on the calendar as it’s likely difficult to find convenient times across every participants’ calendar.
Before & After the meeting
It’s critical to use the WG meeting time as productively as possible. This is especially true when the WG is ramping up, before any experiments have been run.
Here are a few tips for the program manager responsible for leading the meeting:
- Create and share an agenda with goals – Before each meeting the experimentation program manager should create an agenda with specific goals and share it with the WG. This helps ensure that all relevant topics are discussed, including action items that need to be followed up on and new findings that may impact previous plans. The agenda also lets stakeholders know whether they need to attend that specific meeting. For instance, if an agenda is focused on the data science team explaining experimental results to business stakeholders, engineering may decide to skip that meeting.
- Summarize and share meeting notes – The program manager should share summary notes after the meeting. Be sure to include any key decisions made during that meeting, action items that need to be completed, and who’s accountable for those action items. Share these notes with all working group participants, regardless of whether they were present at the meeting or not. These notes can also serve to inform employees outside of the working group about the progress of the ExPr.
- Have an internal data science discussion – Data scientists need business context in order to do their best work. I’ve found that meeting with data scientists who aren’t part of the working group to discuss the meeting notes is an effective way of providing that context. In addition, maintaining an easily accessible log of these notes helps data scientists track decisions over time.
Meeting between Meetings
Occasionally subsets of the WG participants need to meet outside of the WG meeting to complete specific action items. These meetings are where the sausage gets made – where specific details are discussed and determined – and should include the relevant WG participants as well as the individual contributors (ICs) responsible for implementing the work. The WG participants are there to articulate the goals and provide necessary context while the ICs translate this context into specific work to be performed.
For example, suppose the WG determines that log data from certain technology systems must be collected in order to conduct a specific experiment. Representatives from the data science and engineering stakeholder groups might meet to discuss:
- Which specific systems need to be instrumented
- Which user actions will generate log messages
- What attributes the log messages should contain
- Where these logs will be stored
- How the logs can be retrieved for analysis
The WG representatives in these meetings are responsible for sharing status updates with the broader working group. Have the action items been completed? Are tasks on track to be completed? In cases where progress is blocked, the stakeholders should describe what the blockers are, how the groups can get past these issues, and what the new expected timelines are.
Conclusion
In summary, an experimentation program can be driven forward through the establishment of a working group. The working group is composed of key decision makers from stakeholders including business units, data science, and engineering.
The working group meeting is the main tool the working group uses to drive positive business impact through experimentation. The meeting is where participants discuss hypotheses and potential experiments, define and assign specific action items, and share status updates.
The working group and meeting should be led by the experimentation program manager. This role could be played by a data science product manager.
There are a number of ways to improve the working group’s probability of success including explicitly stating and aligning on the group’s goals, establishing a tight feedback loop, developing meeting agendas, sharing summary notes, and having breakout meetings where low-level details are discussed.
In my next post I’ll describe how to define a specific experiment. This experiment definition is the main output the working group generates before implementing and analyzing experiments. If you’d like to be notified when I publish this series, sign up below and I’ll email you each post as I publish them.
Opinions expressed here are my own and do not express the views or opinions of my employers.