Video-based Prediction of Hand-grasp Preshaping with Application to Prosthesis Control

authors: Luke T. Taverne*, Matteo Cognolato*, Tobias Bützer, Roger Gassert, Otmar Hilliges
publication: International Conference on Robotics and Automation (ICRA), Montreal, Canada, May 2019
*Authors contributed equally to this work.


Left: System schematic. As the user reaches for the desired object (a), the forearm-mounted RGB-D video camera (b) and Myo armband (c) stream data to a smartphone application (d). The system uses this data to generate the required grasp-type (in this example, medium wrap). Right: Model diagram for the grasp-type prediction over a grasping sequence with video input.


Among the currently available grasp-type selection techniques for hand prostheses, there is a distinct lack of intuitive, robust, low-latency solutions. In this paper we investigate the use of a portable, forearm-mounted, video-based technique for the prediction of hand-grasp preshaping for arbitrary objects. The purpose of this system is to automatically select the grasp-type for the user of the prosthesis, potentially increasing ease-of-use and functionality. This system can be used to supplement and improve existing control strategies, such as surface electromyography (sEMG) pattern recognition, for prosthetic and orthotic devices. We designed and created a suitable dataset consisting of RGB-D video data for 2212 grasp examples split evenly across 7 classes; 6 grasps commonly used in activities of daily living, and an additional no-grasp category. We processed and analyzed the dataset using several state-of-the-art deep learning architectures. Our selected model shows promising results for realistic, intuitive, real-world use, reaching per-frame accuracies on video sequences of up to 95.90% on the validation set. Such a system could be integrated into the palm of a hand prosthesis, allowing an automatic prediction of the grasp-type without requiring any special movements or aiming by the user.

Accompanying Video


  title={Video-based Prediction of Hand-grasp Preshaping with Application to Prosthesis Control},
  author={Taverne, Luke T. and Cognolato, Matteo and Bützer, Tobias and Gassert, Roger and Hilliges, Otmar},
  booktitle={International Conference on Robotics and Automation (ICRA)},