5,426,970 AI resources indexedWorking toward the world’s largest AI resource index.

Browse resources 13 sections
PDF

A Reparameterization-Invariant Flatness Measure for Deep Neural Networks

The performance of deep neural networks is often attributed to their automated, task-related feature construction. It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples. Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error. For several flatness measures, this correlation has been empirically validated. However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization due to a

By Petzka, Henning, Adilova, Linara, Kamp, Michael, Sminchisescu, CristianResearch paper

Resource information

Authors
Petzka, Henning, Adilova, Linara, Kamp, Michael, Sminchisescu, Cristian
Research area
Artificial intelligence
Research topics
Machine Learning (cs.LG), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Computer and information sciences
Publication date
2019
DOI
10.48550/arxiv.1912.00058
Upstream last modified
Feb 26, 2022
License
arXiv.org perpetual, non-exclusive license
Last record update
Nov 7, 2026

About this resource

The performance of deep neural networks is often attributed to their automated, task-related feature construction. It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples. Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error. For several flatness measures, this correlation has been empirically validated. However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization due to a lack of invariance with respect to reparameterizations. We propose a natural modification of existing flatness measures that results in invariance to reparameterization.