papersTODAY 04:00 UTC
Initial Weight Scale Found to Control Feature Specialization in ReLU Networks
A research paper examines how the scale of a network's initial weights influences which features individual ReLU neurons end up learning during training. The authors argue that this initialization-dependent effect, framed as a hidden gauge, helps explain why trained networks organize representations the way they do.