Scaling laws
A paper that reads the rules of thumb behind bigger AI models the way an economist reads a factory.
- Kind
- Research paper
- Title
- What optimizing labs reveal: scaling laws as production functions
- License
- MIT (code)
- Written with
- Claude
What it is about
AI models get better as they get more parameters, more training data and more computing power. Scaling laws are the curves that describe how much better.
Economists already have a name for a rule that turns inputs into output: a production function. This paper treats scaling laws as exactly that.
What it finds
Labs pick the cheapest mix of model size and data. Because we only see the mixes they chose, their results are a poor guide to how the technology behaves away from those choices.
The same choices do reveal what the labs were aiming for. The paper works out what scaling data can and cannot tell us, measures how easily model size and data stand in for each other, and uses the habit of training small models for longer to estimate how much developers value a compact model.
How it was done
It includes a controlled experiment whose plan was written down before the results were in, and a replication package with the code and the experiment's records.