Studying generalization performance of random feature model through equivalent models

Abstract

Random Feature Model (RFM) has been received significant attention due to its similarity to neural networks and its relatively simple analysis. The training and generalization performances of the RFM has been shown to be equivalent to the noisy linear model under isotropic data assumption in the asymptotic limit. It is possible to analyze the performance of the RFM using the equivalent model. In this work, we focus on studying the generalization performance of the RFM using equivalent models and extending it to anisotropic data. First, we illustrate the regimes where the equivalence with the aforementioned linear model holds and where it does not hold on Fashion-MNIST dataset. Then, we observe a new equivalence for a regime where the aforementioned equivalence does not hold.