Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models
arXiv:2610.39016v1 Announce Type: cross Abstract: Compositional generalization remains a persistent limitation of transformer language models: even when a model learns to combine known facts correctly in-distribution (ID), the same compositional mechanism may fail when applied to out-of-distribution (OOD) knowledge. We investigate this compositionality gap in a controlled two-hop reasoning setting using a GPT-2-like transformer model trained from scratch. Building on prior work identifying a ci…