In late 2017 Andrej Karpathy (Director of AI at Tesla, former Research Scientist at OpenAI) published a blog post called Software 2.0 in which he explained that the growth in use of machine learning and neural networks represent a shift in how software is written. Whereas software 1.0 consists of hand-written rules written by humans in languages like C++ and Python, software 2.0 is written in the language of weights by specifying some goal and using computational resources to search a program space through optimization. According to Karpathy, programmers will split into 2 teams: 2.0 programmers will be responsible for curating, cleaning, and labeling datasets, and 1.0 programmers will focus on maintaining the surrounding tools, visualizations, and infrastructure.
Although we’re not quite there yet, it’s fascinating to think through what this transition could mean for software companies and the technology industry as a whole.
Bootloading a Company with Software 1.0
One possible scenario is a future where human engineers are used to "bootstrap" an application or system. By bootstrap I mean that software developers will develop applications much as they do today: by designing interfaces, using frameworks, and writing testing suites.
The most critical part of this bootloading procedure will be properly instrumenting systems with "sensors" to capture and store telemetry. In subsequent iterations of the system this telemetry data will be used to learn the optimal modules and subcomponents of that application. So in some sense every engineer responsible for initially implementing the system will be a data engineer, responsible for instrumenting their hand-written application with sensors to capture appropriate sensor data. This shift will lead to the proliferation of new programming paradigms and development tools to automatically generate, expose, and capture meaningful metrics. Such data points will measure application performance and simplify the procedure of learning an optimal program.
Learning to Code with Software 2.0
Once the initial application has been developed and its sensor data generated and stored, engineers will focus on using the data to learn programs. Rather than design the software, engineers will need to design the model’s objectives i.e. the goals of the system. That is, what objective should the optimization algorithms responsible for learning optimize for? By designing a system’s goals up front, engineers will enable optimization algorithms to learn the best program for the job by limiting the search space.
Designing this objective will be easier in some cases than in others. As a company’s impact grows and its products influence larger groups of people, designing objectives will become very difficult, error-prone, and messy. For instance, how do you efficiently design learning goals for people who share different values and cultures across international borders? ML product managers at large companies like YouTube and Facebook have learned that unpredictable issues can arise when you optimize for simple objectives like profit or engagement. And there are numerous other examples where optimizing for a certain metric has led to unintended negative outcomes, despite the positive intentions of the talented and experienced individuals responsible for designing the objective.
The Time to Prepare for Tomorrow is Today
What can we do today to prepare for tomorrow? I recommend that every software engineer learn a thing or two about running machine learning systems in production.
But know that you don’t need to master learning algorithms to effectively contribute. Perhaps you’re interested in front-end development, in which case you might help create the data visualization systems of tomorrow. Or maybe you’re a hardware fanatic and your skills and passion can be directed towards the challenges of federated learning or optimized inference. If DevOps is your forté, know that there will be many issues in operating and orchestrating these systems of tomorrow (and there is a lot of low hanging fruit in the MLOps space today).
While of spoken only of technical roles, know that there will be lots of work for non-technical folks as well. If you’re looking to transition into the space and want more advice, leave comment below. I have plenty of ideas : )
I’m currently attending Lambda School and learning a more general approach – the entire data science/ML pipeline. Most of your recommendations are for people with previous experience. Any thoughts for someone just breaking into the field?
Hi Michael. My blog is aimed at helping data scientists, ML engineers, and ML Product Managers who are already in the field. I encourage you to check out why I started my blog. If you’re looking to get hired, check out some advice I gave on that topic.