Compositionality and Disentanglement: Structured Representations in Simple Neural Models

Abstract

Název práce: K  : S   ý ý  Autor: M A Pracoviště: Ú í    Vedoucí práce: . RND. P P, P.D., Ú í    Abstrakt: Z ,     ý í  č í   ý  (í )   čí  (ý ). V  ří    ů:    ří. K  í       čí . N ří  í ů  . P ěří  í   čí     žý   ěí ý . D , ž     ší ňí,    říě, ž   čí . P   í í   í ů (),  í,   í   ý í    ě ě   ý ů. S čí, ž   í   í, ž ý  č í ý...

Title: Compositionality and Disentanglement: Structured Representations in Simple Neural Models Author: Michal Auersperger Department: Institute of Formal and Applied Linguistics Supervisor: doc. RNDr. Pavel Pecina, Ph.D., Institute of Formal and Applied Linguistics Abstract: We study conditions that encourage the emergence of structured neural represen- tations in communication games with discrete channels and in autoencoders with continuous channels. Across both settings, we examine two pressures: compres- sion, implemented through message length penalties and bounded activations, and training pressure, introduced through agent resetting and dropout. In signaling games, we propose grammar-based compositionality metrics derived from proba- bilistic context-free grammar induction and show that structured protocols support better generalization under constrained training conditions. For continuous repre- sentations, we introduce a vocabulary-based reconstruction measure that evaluates whether latent fragments can be recombined into valid codes. Experiments in con- trolled settings suggest that compression and training pressures promote represen- tations whose parts can be reused more systematically. Keywords: compositionality, generalization, emergent languages, disentan- glement iii