Part three of a three-part series on the actual work inside artificial intelligence. Part one describes the AI shed: factory and research; part two, what philosophers do in AI. In the usual picture, the philosopher asks questions and the engineer solves problems. Looked at closely, the last decade of artificial intelligence does not support that division. Several of the field’s central technical debates are, to a large extent, philosophical problems, even though they are published on arXiv, formulated in equations and argued over by people who never took a course in epistemology. This third part walks through that technical heart of the shed and the unnamed philosophy it contains.

The trade in the factory

It is worth starting with what is not philosophy. Most engineering work in the industry is building and operating: data engineering at petabyte scale, distributed training infrastructure, post-training and labeling operations, evaluation engineering, efficient inference, operational safety, product integration. In part one I described that chain. Its central demand is reliability: that a system of enormous complexity work every day, with constant hardware failures, millions of users and costs measured in hundreds of millions of dollars. That trade is engineering in the most classical sense, and its value does not depend on any philosophical reflection. But it is also where decisions with normative consequences are made, often without anyone noticing: which data is included and which is not, which metric defines success in an evaluation, which threshold triggers a safety filter. Each of those decisions is a value judgment disguised as a parameter.

The research milestones

The lab of the last decade has few theoretical milestones, and very concrete ones. The Transformer architecture (Vaswani et al., 2017) is the basis of almost everything that came afterward. Its central idea, the attention mechanism, lets every word in a text relate directly to all the others, and it parallelizes very well on available hardware. That last, more prosaic property explains its success as much as the first. Scaling laws (Kaplan et al., 2020) showed that the performance of these models improves predictably, following power laws, as parameters, data and compute increase. Two years later, Chinchilla (Hoffmann et al., 2022) corrected the optimal ratio between model size and amount of data: the models of the time were undertrained. Richard Sutton had summed up the spirit of the era in The Bitter Lesson (2019): in the long run, general methods that leverage growing compute beat those that try to encode human knowledge. It is a bitter lesson for anyone who devoted a career to the latter, and a strong epistemological thesis about the relationship between knowledge and search. Reinforcement learning from human preferences (Christiano et al., 2017; InstructGPT, 2022) turned models that predict text into assistants that follow instructions. More recently, inference-time compute, which lets the model reason for longer before answering, opened a second axis of scaling.