Long Sequence Hopfield Memory
Keywords: Hopfield network, associative memory, sequence modeling, motor neuroscience
TL;DR: We propose a model for sequence memory based on asymmetric Modern Hopfield Networks, analyze its capacity, verify it with numerical simulation, and demonstrate its usefulness in robustly recalling correlated sequences of patterns.
Abstract: Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules. However, these networks suffer from limited sequence capacity (maximal length of the stored sequence) due to interference between the memories. Inspired by recent work on Dense Associative Memories, we expand the sequence capacity of these models by introducing a nonlinear interaction term, enhancing separation between the patterns. We derive novel scaling laws for sequence capacity with respect to network size, significantly outperforming existing scaling laws for models based on traditional Hopfield networks, verify these theoretical results with numerical simulation, and demonstrate their usefulness in overlapping patterns. Finally, we describe a biologically-plausible implementation, with connections to motor neuroscience.
Submission Number: 24
Loading