Last week I wrote that companies fall into one of two groups when it comes to machine learning and data science. The first class contains companies where ML/DS is a core competency. Without their machine learning models, datasets, and expertise, these companies would look very different in terms of their product and service offerings, their technical philosophies, and staffing. The second, far larger class contains companies where ML/DS is not at the core of their existence. These companies vary in their levels of experience applying data science, the types of data science roles they have on staff, and the number of employees they have within these roles. What they have in common, however, is that data science is an add-on capability. These companies want to make their operations more efficient, or they want to add a new feature to their product to increase revenue, or they just want to be tell their investors "hey look, we’re doing machine learning too!"
I don’t see either of these groups as being objectively better than the other. Whether you want to work for companies in group 1 vs group 2 is purely a matter of personal opinion. I’ve worked for companies in both groups and can say confidently that there are pros and cons to both. In fact, I can make an argument for working at companies that have data science teams just for the sake of marketing that capability. One’s satisfaction is a function of his or her expectations, goals, and beliefs.
While neither group is objectively "better", they are quite different. Arguably the biggest difference between companies in these groups is the level of influence data science has on the company. For the rest of this issue, I want to focus on the importance of influence within data science teams at companies where ML/DS is not a core competency.
If you’re at one of these companies, the biggest challenge you’ll face is getting key business stakeholders and decision makers to take a chance on data science. These companies have gotten to where they are today without relying on fancy machine learning algorithms. They likely have several key employees that have the final say on the most important decisions. These employees have likely worked there for many years and seen many fads come and go. They’ve also seen many employees come and go. Their livelihoods are very much tied to the success of the business. I won’t go further – my point is that these people need to have a very good reason to introduce any amount of risk into their organization.
If you’re on a data science team at one of these companies, you need the ability to influence these folks if you want them to adopt your solutions. How do you develop that influence?
One way is to have a foot at the table with the most senior decision makers. This could mean having a chief data scientist who is involved in shaping business strategy at the highest levels. This person has the difficult job of building trusting relationships with people from around the business, deeply understanding their needs, and communicating these needs to the data science team. It takes a true mix of business savvy, marketing skills, and technical knowledge to play this role.
What if you don’t have a C-level data science leader? While it’s not impossible to make progress here, I would say that the task is more difficult. I recommend learning as much as you can about different areas of the business and identifying key pain points and opportunities. Learn to make a business case for why machine learning or data science is necessary to solve these challenges. Be able to talk in terms of dollars and cents. And be prepared to be in it for the long haul. At the end of the day, you’re trying to convince someone to risk their reputation on stochastic algorithms that can fail for many different reasons. It shouldn’t be too shocking that they’d prefer their problems solved by an if-else statement.
Here’s what I’ve been reading/watching/listening to recently:
- Overlapping Experiment Infrastructure: More, Better, Faster Experimentation – At Google experimentation is used to evaluate "almost every change that potentially affects what users experience" including not only modifications to a user interface but also testing different machine learning algorithm changes that might affect ranking or content selection. This paper describes Google’s "overlapping experiment infrastructure" as well as the associated tools and educational processes to support more, better, and faster experimentation. While the paper is specific to Google web search, the lessons contained within any company that wants to evaluate changes with empirical data.
- How to Build an Experiment Pipeline from Scratch – Whereas the previous paper discusses how to optimize an already mature experiment infrastructure, this blog post describes a quick and resourceful way to bootstrap an experimentation framework. The author describes the approach he took to build an experimentation pipeline at Shopify to measure Marketing decisions, for example the benefits of a recommender system that generates personalized blog post suggestions. The post includes his step-by-step approach for understanding the business need, getting stakeholder buy-in, technical planning, and implementation.
- How to Use Quasi-experiments and Counterfactuals to Build Great Products – While A/B tests are the gold standard method for determining causality, it’s sometimes not possible to set up experiments for reasons including lack of tooling, lack of time, or ethical concerns. In these situations, we can look to other methods on the "evidence ladder for causal inference" including quasi-experiments and counterfactuals. While these methods are great when fully random experimentation isn’t possible, keep in mind that it’s harder to compute confidence intervals i.e. there’s more uncertainty involved.
- How should our company structure our data team? – Should your data team be organized functionally or by division (different areas of the business)? Do you adopt a center of excellence model, where all data people reside on one team, or an embedded model where people report to individual business units? As this post describes, these are difficult questions that depend on the specific context of your company. The writer details how his team evolved their organization over 5 stages in 3 years and the pros and cons of each.
- What is a Feature Store? – I’ve consumed many blog posts and conference talks that mention the benefits of feature stores, but none do so as comprehensively as this blog post from Tecton. According to the post, feature stores enable operational machine learning (i.e. ML-driven real time applications like recommender systems, fraud detection, and personalization) by allowing teams to solve common data management problems. There are 5 components of a modern feature store: transformation, storage, serving, monitoring, and feature registry. Although the post is clearly marketing collateral, I think it’s valuable for data science leaders encountering data management challenges.
That does it for this week’s issue. If you have any thoughts, I’d love to hear them in the comment section below!