papersTODAY 04:00 UTC
Study Finds Structured Features Overfit Where Random Gaussian Features Grok
New research contrasts how over-parameterized ridge regression behaves with structured versus unstructured random feature maps. Prior work by Xu, Vardi and Safran proved that random Gaussian features grok, with the gap between memorization and generalization widening as the weight decay parameter shrinks. The authors report that structured features instead overfit in the regime where random features exhibit grokking.