Below is an interview with Eric Colson, Emeritus Chief Algorithms Officer, Stitch Fix Data Science.
Our interview with Eric Colson is part of our interview series with the creators of some of the top ML resources we’ve come across online.
Eric’s top ML resource, The power of the full-stack data science generalist and the perils of division of labor through function, does a great job explaining how teamwork can impact effectiveness in data science.
1. How did you get started working on machine learning? How have you progressed through the ML space?
My undergrad was in economics. More specifically, micro-economics was an area of particular focus for me. I was intrigued by the way a company could leverage information for a competitive advantage (e.g. by better estimating demand, by optimizing prices, through better management of inventory, or more efficient investments in marketing, etc). The gains in efficiency benefit both the the consumers and company alike — Such a noble way to compete! Through my economics program I eagerly learned the theory behind framing such problems.
Alas, if only it were practical! After entering the workforce, I learned that economics was not the only thing needed in order to implement solutions to microeconomic problems. One also needed statistics to empirically solve the various pieces (fitting a curve to a set of messy data, controlling for confounding factors, understanding variation, etc). Also, one needed computer science to implement a solution — typically an algorithm that could be deployed within an existing transaction processing system.
I was able to acquire some of the requisite skills on the job – though only enough to get the job done (ie enough to be dangerous!!). I later acquired more fundamentals when I returned to grad school. To this day, I still believe that diversity in skills and training – especially in the domains of stats, CS, and econ/business – is essential for success in data science / ML.
2. What differentiates successful industry ML projects from unsuccessful projects?
One mistake I see companies make is to put too much effort into up-front design. There is inherent uncertainty in ML. There are often surprises in the data (in quality, in interpretation, in volume, and even in processing requirements). In addition, training ML models requires a lot of trial and error (in model selection, feature selection, tuning hyper parameters, etc). Further, the resulting model typically comes with it’s own set of implementation challenges that are difficult to anticipate (latency requirements, reliability, scaling, etc). These things make it extremely difficult to design a solution up-front. Rather, one needs to evolve a solution as new information is revealed through iteration and learning.
Yet, many companies use a planning process that doesn’t allow variation from up-front designs. A more effective approach is to provide business context and a general goal. Then embrace a test and learn strategy. It’s quite common for the resulting solution to look very different from what was originally envisioned. And, this is a good thing! The learnings that occur along the path to a solution typically could not have been anticipated and the resulting system benefits greatly from the new knowledge.
Certainly, caveats apply here. Domains that are well-understood and have less uncertainty benefit from more up-front design.
3. What advice do you have for ML practitioners who are struggling to build machine learning solutions into products?
Look at your processes. Do they allow for iteration and learning or are they based on up-front design? What is an organizational/management challenge of running effective machine learning teams and what’s the best way to tackle this challenge?
It’s important to discern whether you need to “Organize to Execute” vs. “Organize to Learn”. I had noticed the difference in my career but lacked the vocabulary to describe it until I read Amy Edmondson’s book, Teaming. If requirements are clear and designs can be done up-front, then you want to organize the team to execute as efficiently as possible and to not vary from requirements. When this is the case, teams of narrow specialists will benefit from the division of labor and can efficiently deliver on the requirements.
However, when the solution is ambiguous with a lot of uncertainty then you need to organize to learn. This is very typical for ML initiatives and merits smaller teams of individuals with more general skills in order to reduce coordination costs and allow for faster iteration and learning.
The distinction may seem obvious. However, many companies orient their ML teams towards execution only. They do so out of familiarity (many manufacturing process work this way and become the default model for running other teams) or because they prefer the clarity of up-front design vs. the daunting ambiguity of a learn-as-you-go process. But there is often no getting around the fact that the learning is an essential part of the process. It’s best to embrace it and set the team up for success through the right organizational strategy.
You can follow Eric Colson on Twitter here.