Sketched symbol recognition with auto-completion

Abstract

Sketching is a natural mode of communication that can be used to support communication among humans. Recently there has been a growing interest in sketch recognition technologies for facilitating human-computer interaction in a variety of settings, including design, art, and teaching. Automatic sketch recognition is a challenging problem due to the variability in hand drawings, the variation in the order of strokes, and the similarity of symbol classes. In this paper, we focus on a more difficult task, namely the task of classifying sketched symbols before they are fully completed. There are two main challenges in recognizing partially drawn symbols. The first is deciding when a partial drawing contains sufficient information for recognizing it unambiguously among other visually similar classes in the domain. The second challenge is classifying the partial drawings correctly with this partial information. We describe a sketch auto-completion framework that addresses these challenges by learning visual appearances of partial drawings through semi-supervised clustering, followed by a supervised classification step that determines object classes. Our evaluation results show that, despite the inherent ambiguity in classifying partially drawn symbols, we achieve promising auto-completion accuracies for partial drawings. Furthermore, our results for full symbols match/surpass existing methods on full object recognition accuracies reported in the literature. Finally, our design allows real-time symbol classification, making our system applicable in real world applications.