PHOTON (Parallel Hierarchical Operation for TOp-down Networks) is a hierarchical autoregressive model that replaces the horizontal token-by-token scanning of Transformers with vertical, multi-resolution context scanning.

The model features a hierarchy of latent streams: a bottom-up encoder compresses tokens into low-rate contextual states, while lightweight top-down decoders reconstruct fine-grained token representations in parallel. It also introduces recursive generation, which updates only the coarsest latent stream and eliminates bottom-up re-encoding.

Experimental results show that PHOTON outperforms competitive Transformer-based language models in the throughput-quality trade-off, particularly in long-context and multi-query tasks. It reduces decode-time KV-cache traffic, yielding up to 1000x higher throughput per unit memory.

Deep artificial neural networks struggle with learning from non-stationary data streams, leading to forgetting and loss of plasticity in continual learning scenarios.

Current continual learning approaches have focused on either stabilizing representations of past tasks or promoting plasticity for future learning. However, biological neural networks exhibit representational drift, where responses associated with stable behaviors gradually change over time, suggesting this might be a key property for continual learning in biological systems.

This research explores how linking representational drift to continual learning could inform artificial systems. Drift may reflect a mixture of homeostatic turnover and learning-related synaptic plasticity, potentially offering insights for improving continual learning approaches in artificial systems.

This paper introduces a biologically inspired, multilayer neural architecture composed of Rectified Spectral Units (ReSUs). Each ReSU projects a recent window of its input history onto canonical directions obtained via canonical correlation analysis (CCA) of past-future input pairs, and then rectifies either its positive or negative component.

By encoding canonical directions in synaptic weights and temporal filters, ReSUs implement a local, self-supervised algorithm for progressively constructing increasingly complex features. This offers a biologically grounded, backpropagation-free paradigm for constructing deep self-supervised neural networks.

When a two-layer ReSU network was trained in a self-supervised regime on translating natural scenes, first-layer units developed temporal filters resembling those of Drosophila post-photoreceptor neurons, and second-layer units became direction-selective, suggesting ReSUs are promising for modeling biological sensory circuits.