Training shapes the curvature of shallow neural network representations
Keywords: Riemannian geometry, neural networks, curvature
TL;DR: We study how training shapes the Riemannian geometry induced by neural network feature maps.
Abstract: We study how training shapes the Riemannian geometry induced by neural network feature maps. At infinite width, shallow neural networks induce highly symmetric metrics on input space. Feature learning in networks trained to perform simple classification tasks magnifies local areas and reduces curvature along decision boundaries. These changes are consistent with previously proposed geometric approaches for hand-tuning of kernel methods to improve generalization.
7 Replies
Loading