Nouveau Recherche PDF, HTML, DOCX et bien d'autres formats.

Prépublication · 2026

Structure--Residual Inductive Biases for Language Model Pretraining

Hai Huang, Bartosz Żuk et al. — Monde

Source : arXiv

Résumé

Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regular…

Actions

Citation