TU Dresden
A joint project with Uni Leipzig DE EN

Surg-Morph: Neuromorphic Surgical Video Processing on the SpiNNaker2 Platform

Mentored by Christian Mayr & Stefanie Speidel
at Chair of Highly-Parallel VLSI Systems, TU Dresden/National Center for Tumor Diseases (NCT), UKD Dresden

In surgical data science, the need for data processing in real-time is of utmost importance, to guarantee that an online assistance, such as visualization of navigation information or predictions of complications, during surgery can be provided. The major source of information during surgery is video data from, for example, surgical endoscopes. To enable online assistance, real-time and energy-efficient methods for 2D-image and video processing (e.g. semantic segmentation, workflow analysis, object detection, … ) as well as 3D-scene understanding (e.g. 3D reconstruction, camera localisation, object registration) are essential. This is currently limited by the availability of AI hardware that is sufficiently performant at real time.

Neuromorphic computation is considered as one of the most promising approaches to overcome the speed and power limitations of current DNN hardware. The large-scale SpiNNaker2 neuromorphic system employs a massive amount of ARM M4f microcontrollers (153 per chip, 10Mio in overall machine) that are embedded in a slim, low-latency communication fabric inspired from neuronal connections in the brain. It is a generally programmable computing substrate, but greatly differs from classical architectures like CPUs or GPUs, allowing for more distributed and asynchronous processing and simulation approaches.

Thus, the SpiNNaker2 architecture with its highly parallel and distributed processing architecture and focus on millisecond latency is a promising candidate for real-time, surgical data processing at massive scales, at increased energy efficiency. A tentative workplan could be thus: Establish a baseline model (DNN ,etc) for real-time surgical data processing (e.g. endoscopic video processing). Develop and implement a mapping strategy for the baseline network on SpiNNaker2. Refine baseline with neuromorphic inspirations to fully utilize the SpiNNaker2 machine.

Work Environment

You will be working at the Chair of Highly-Parallel VLSI Systems and Neuromicroelectronics (HPSN) Dresden and the National Center for Tumor Diseases (NCT) Dresden. Your primary workplace (HPSN) is located at the main campus of TU Dresden. On site, you will work with the largest neuromorphic and real-time AI machine currently existing, the 10Mio core, 16 rack SpiNNaker2 machine. Research at HPSN focusses on real-time AI, as well as neuromorphic hardware and algorithms, plus miscellaneous circuit design topics. The 2nd workplace will be at the National Center for Tumor Diseases (NCT) Dresden is the associate partner of the project. The NCT Dresden is located at the medical campus and offers a unique research platform including an experimental operating room and a novel simulation room for robot-assisted surgery. The group researches applied machine learning methods for robot-assisted surgery and surgical data science.

Prerequisites

Master’s Degree (or equivalent) in computer science, electrical engineering, applied mathematics or related fields of expertise.

  • Very good programming skills (e.g. C++, Python)
  • Excellent skills and practical experience in one or more of the following research areas is beneficial:
    • Machine Learning, esp Hardware/Embedded Systems
    • Computer Vision
    • Neuromorphic Systems
  • Ability to collaborate well in an interdisciplinary environment