Too Big to Think

This is an shortened version of the Oral Presentation of my paper Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers https://arxiv.org/abs/2506.09099
I gave an oral presentation of this paper at the TTODLer-FM workshop at ICML 2025: https://icml.cc/virtual/2025/workshop/39957

If you’re interested in seeing the recording of that presentation, you can click the SlidesLive Video Button for the 9:30am timestamp (Under Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers)

Here is the paper’s abstract:
The relationship between memorization and generalization in large language models (LLMs) remains an open area of research, with growing evidence that the two are deeply intertwined. In this work, we investigate this relationship by pre-training a series of capacity-limited Transformer models from scratch on two synthetic character-level tasks designed to separately probe generalization (via arithmetic extrapolation) and memorization (via factual recall). We observe a consistent trade-off: small models extrapolate to unseen arithmetic cases but fail to memorize facts, while larger models memorize but fail to extrapolate. An intermediate-capacity model exhibits a similar shift toward memorization. When trained on both tasks jointly, no model (regardless of size) succeeds at extrapolation. These findings suggest that pre-training may intrinsically favor one learning mode over the other. By isolating these dynamics in a controlled setting, our study offers insight into how model capacity shapes learning behavior and offers broader implications for the design and deployment of small language models.