GraspLook: A VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment

Polina Ponomareva, Daria Trinitatova, Aleksey Fedoseev, Ivan Kalinov, Dzmitry Tsetserukou

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Citation (Scopus)

Abstract

The teleoperation of robotic systems in medical applications requires stable and convenient visual feedback for the operator. The most accessible approach to delivering visual information from the remote area is using cameras to transmit a video stream from the environment. However, such systems are sensitive to the camera resolution, limited viewpoints, and cluttered environment bringing additional mental demands to the human operator. The paper proposes a novel system of teleoperation based on an augmented virtual environment (VE). The region-based convolutional neural network (R-CNN) is applied to detect the laboratory instrument and estimate its position in the remote environment to display further its digital twin in the VE, which is necessary for dexterous telemanip-ulation. The experimental results revealed that the developed system allows users to operate the robot smoother, which leads to a decrease in task execution time when manipulating test tubes. In addition, the participants evaluated the developed system as less mentally demanding (by 11 %) and requiring less effort (by 16%) to accomplish the task than the camera-based teleoperation approach and highly assessed their performance in the augmented VE. The proposed technology can be potentially applied for conducting laboratory tests in remote areas when operating with infectious and poisonous reagents.

Original languageEnglish
Title of host publication2021 20th International Conference on Advanced Robotics, ICAR 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages166-171
Number of pages6
ISBN (Electronic)9781665436847
DOIs
Publication statusPublished - 2021
Event20th International Conference on Advanced Robotics, ICAR 2021 - Ljubljana, Slovenia
Duration: 6 Dec 202110 Dec 2021

Publication series

Name2021 20th International Conference on Advanced Robotics, ICAR 2021

Conference

Conference20th International Conference on Advanced Robotics, ICAR 2021
Country/TerritorySlovenia
CityLjubljana
Period6/12/2110/12/21

Fingerprint

Dive into the research topics of 'GraspLook: A VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment'. Together they form a unique fingerprint.

Cite this