Two-hand on-skin gesture recognition: a dataset and classification network for enhanced human-computer interaction

Abstract

Gestural interaction is an increasingly utilized method for controlling devices and environments. Despite the growing research on gesture recognition, datasets tailored specifically for two-hand on-skin interaction remain scarce. This paper presents the two-hand on-skin (THOS) dataset, comprising 3096 labeled samples and 92,880 frames from three subjects across nine gesture classes. The dataset is based on hand-specific on-skin (HSoS) gestures, which involve direct contact between both hands. We also introduce THOSnet, a hybrid model leveraging transformer decoders and bi-directional long short-term memory (BiLSTM) for gesture classification. Evaluations show that THOSnet outperforms standalone transformer encoders and BiLSTMs, achieving an average test accuracy of 79.31% on the THOS dataset. Our contributions aim to bridge the gap between dynamic gesture recognition and on-skin interaction research, offering valuable resources for developing and testing advanced gesture recognition models. By open-sourcing the dataset and code through https://github.com/ege621/thos-dataset, we facilitate further research and reproducibility in this area.