Pade neurons for efficient neural models

Abstract

Neural networks commonly employ the McCulloch-Pitts neuron model, which is a linear model followed bya point-wise non-linear activation. Various researchers havealready advanced inherently non-linear neuron models, such asquadratic neurons, generalized operational neurons, generativeneurons, and super neurons, which offer stronger non-linearitycompared to point-wise activation functions. In this paper, weintroduce a novel and better non-linear neuron model calledPad e neurons (Paons), inspired by Pad e approximants.Paonsoffer several advantages, such as diversity of non-linearity, sinceeachPaonlearns a different non-linear function of its inputs,and layer efficiency, sincePaons provide stronger non-linearity inmuch fewer layers compared to piecewise linear approximation.Furthermore,Paons include all previously proposed neuronmodels as special cases, thus any neuron model in any networkcan be replaced byPaons. We note that there has been a proposalto employ the Pad e approximation as a generalized point-wiseactivation function, which is fundamentally different from ourmodel. To validate the efficacy of Paons, in our experiments,we replace classic neurons in some well-known neural imagesuper-resolution, compression, and classification models basedon the ResNet architecture withPaons. Our comprehensiveexperimental results and analyses demonstrate that neural mod-els built byPaons provide better or equal performance thantheir classic counterparts with a smaller number of layers. The PyTorch implementation code forPaonis open-sourced at https://github.com/onur-keles/Paon