TU Dresden
A joint project with Uni Leipzig DE EN

GetInTouch – Prediction of Hand Gestures for Intuitive Human-Machine Interaction

Mentored by Frank Fitzek, Stefanie Speidel, Merve Sefunç (industry partner at Mimetik)
at Deutsche Telekom Chair of Communication Networks, TU Dresden

Hand gestures are an integral part of an intuitive human-machine interaction in different domains such as industry or medicine. To enable such an interface, different methods in the field of perception, real-time interpretation based on machine learning, as well as context-aware action are required.

The PhD research topic focuses on the recognition and prediction of hand gestures and tasks based on a novel data glove as sensory input. Here the collaboration with Mimetik will take place in order to exploit their data glove (see Figure 1). The use case in this scenario is human-machine interaction for collaborative work scenarios in industry and medicine in the real or virtual world (Metaverse), it will be conducted in close collaboration with Mimetik UG who will provide the data glove.

Data glove of Mimetik. We see the hands of a worker wearing the light-blue gloves while assembling some machinery. © Mimetik

Figure: Data glove by Mimetik

The research challenges that are crucial in this context are for example: i.) How can we generate a representative data set including edge cases for this scenario?; ii.) How can online recognition and prediction be realized using recent advances in machine learning such as transformers or recurrent networks?; iii.) What additional sensory input can be used to enhance accuracy and robustness?; iv.) How can we enable a comparative evaluation framework in the context of an online challenge to create impact?

The research outcome is focusing on: i.) carrying out research in the aforementioned field; ii.) producing high quality publications and annotated data; iii.) foster collaboration with the startup ecosystem as proclaimed in the SECAI research proposal; iv.) inspire the SECAI team with reusable outcome in form of embedded prediction on data gloves.

Work Environment

You will be working with Deutsche Telekom Chair of Communication Networks and the Centre for Tactile Internet with Human-in-the-Loop (CeTI). You will collaborate within a larger interdisciplinary team to discuss and embed your research work. Furthermore, the start-up Mimetik serves as industrial partner and will provide the data glove as well as expertise regarding hardware integration. An additional partner of the project will be the National Center for Tumor Diseases (NCT) Dresden regarding the medical use case. The NCT offers a unique research platform including an experimental operating room and a novel simulation room for robot-assisted surgery.

Prerequisites

  • Master’s Degree (or equivalent) in electrical engineering, computer science, applied mathematics or related fields of expertise
  • Very good programming skills (e.g. C++, Python)
  • Excellent skills and practical experience in one or more of the following research areas is beneficial:
    • Machine Learning
    • Robotics
    • Computer Vision
    • Communication Networks
    • Electronics
  • Ability to collaborate well in an interdisciplinary environment