One-Shot Neural Architecture Search via Novelty Driven Sampling

Abstract

One-Shot Neural architecture search (NAS) has received wide attention due to its computational efficiency. Many One-Shot NAS methods use the validation accuracy based on the supernet as the stepping stone to search the best performing architecture through a bilevel optimization manner, assuming this validation accuracy approximates to the test accuracy after re-training. However, increasing research has found that there is no positive correlation between the above validation accuracy and test accuracy for these One-Shot NAS methods, and this reward based sampling for supernet training also entails the rich-get-richer problem. To handle this deceptive problem, this paper presents a new approach, Efficient Novelty-driven Neural Architecture Search (EN2AS), to sample the most abnormal architecture to train the supernet. Specifically, a single-path supernet is adopted, and only the weights of the single architecture sampled by our novelty search are optimized in each step to reduce the memory demand greatly. Extensive experiments demonstrate the effectiveness and efficiency of our novelty search based architecture sampling method.

Miao Zhang

Professor @ Harbin Institute of Technology

My research interests include Neural Architecture Search(NAS), Bayesian Optimization, High-dimensional Data, and Evolutionary Algorithm.

Shirui Pan

Professor and ARC Future Fellow

My research interests include data mining, machine learning, and graph analysis.