A new arXiv preprint introduces ZAPS, a technique for selecting zero-cost proxies that estimate how well a neural network architecture will perform before any training takes place. Because evaluating candidates normally requires full training runs, such proxies could make architecture search far cheaper. The paper focuses on how to identify which proxy to use for a given search task.
Researchers propose SG-Blend, an activation function that interpolates between an improved version of Swish and GELU rather than relying on either alone. The work addresses the fact that Swish was found through architecture search on vision tasks while GELU became standard in transformer language models, leaving neither optimal across domains. The paper is a revised cross-listing on arXiv.
A newly posted arXiv paper addresses the heavy compute cost of neural architecture search when performance must be measured with leave-one-subject-out evaluation, a protocol used to gauge how subject-based classifiers generalise to unseen individuals. Because a fully nested search would require training a separate architecture for every held-out subject, the authors propose a leakage-free procedure that avoids that expense. The method is aimed at subject-dependent classification settings where per-subject data splits are standard.
Researchers have introduced DiffLUT-Net, a framework that trains neural networks made of FPGA lookup tables directly through differentiable methods rather than converting pretrained quantized models. The approach also learns the connectivity structure of the LUT network, aiming to make hardware-efficient inference on FPGAs more effective. The work is available as a paper on arXiv.