Towards mobile 3D telepresence using had-worn devices and dual-purpose screens

Abstract

Head-mounted displays and augmented reality headsets are emerging as the future of human-computer interaction. Such devices can display high resolution 3D images and use on-board cameras to capture the surroundings of the user. However, capturing the user who is wearing the device to facilitate 3D telepresence is not possible with such headsets. Here we propose and demonstrate a new integrated platform to provide mobile 3D telepresence experience using a head-worn device and a dual-purpose passive screen. At the core of this telepresence architecture, we use a portable multi-layered passive screen which facilitates the stereoscopic 3D display using a pair of head-worn projectors and at the same time, captures the multi-perspective views of the user on a head-worn camera through reflections of the screen. The screen contains retroreflective material for stereo image display and an array of convex mirrors for 3D capture. The 3D telepresence is demonstrated using an experimental setup where a local-user wearing the developed head-worn device perceives the 3D images on the dual-purpose screen, while the captured perspective views of user-1 are rendered as stereo viewpoints and showed to the user-2 on a virtual reality headset.