📅 Date: Monday, July 07, 2026
🕚 Time: 11:00 – 12:00
Location: Heinzel Seminar Room, Office Building West
Speaker: Christos Thrampoulidis (University of British Columbia)
Title: Implicit geometry of deep representations: A log-bilinear softmax view
Abstract:
Training data determines what neural networks can learnbut can we predict the geometry and semantics of learned representations directly from data statistics? I will describe a framework that addresses this question for sufficiently large, well-trained neural networks by viewing them through the coarse but predictive abstraction of log-bilinear softmax models. Within this framework, we show how label imbalance shapes representation geometry and, for language models, how word and context representations organize into semantic structures governed by a sparse-plus-low-rank decomposition of co-occurrence statistics. I will also describe recent progress on the gradient-descent dynamics and implicit bias of softmax models, which provides a link between cross-entropy training and the emergence of neural-collapse-type geometries.
This is based on a series of works on neural collapse, implicit bias, and the dynamics of cross-entropy loss, including:
[1] NeurIPS 2022: https://arxiv.org/abs/2208.05512
[2] COLM 2024: https://arxiv.org/abs/2408.15417
[3] NeurIPS 2024: https://arxiv.org/abs/2402.18551
[4] https://arxiv.org/abs/2505.08348
[5] https://arxiv.org/abs/2512.04006
[6] ICML 2026: https://arxiv.org/abs/2605.23087
[7] https://openreview.net/pdf?id=2xbulaghSN