Implications of node selection in decentralized federated learning

Abstract

Decentralized Federated Learning (DFL) offers a fully distributed alternative to Federated Learning (FL). However, the lack of global information in a highly heterogeneous environment negatively impacts its performance. Node selection in FL has been suggested to improve both communication efficiency and convergence rate. In order to assess its impact on DFL performance, this work evaluates node selection using performance metrics. It also proposes and evaluates a time-varying parameterized node selection method for DFL employing validation accuracy and its per-round change. The mentioned criteria are evaluated using both hard and stochastic/soft selection on sparse networks. The results indicate that the bias associated with node selection adversely impacts performance as training progresses. Furthermore, under extreme conditions, soft selection is observed to result in higher diversity for better generalization, while a completely random selection is shown to be preferable with very limited participation.