Human-centered Control

The human-inspired technological developments enable innovative control applications which are difficult to achieve with traditional control methods. In human-in-the-loop systems, for instance, employing classical system identification techniques for model derivation is challenging because of the complexity and probabilistic nature of human behavior. Thus, our research focuses on developing novel concepts of automatic control for stochastic systems, based on data-driven human models with uncertainties. In addition to advancements in control theory and machine learning techniques, we employ neuroscientific methodologies to understand how the human behavior is controlled by the CNS which are then explicitly interfaced with various control applications such as pHRI.

Current topics:

Related projects:

Current topics:

Human-centred control in clinical applications

Researcher: Satoshi EndoArmin Lederer

Motivation

Our research efforts centre on a continuous, non-invasive monitoring system for patients with movement deficits, and patient-specific adaptive rehabilitation/healthcare technologies. Neurological disorders such as Parkinson’s disease (PD) has complex phenomenology not only across patients but also within themselves across time. Thus, continuous and non-invasive monitoring of clinical states of the patients during daily activities could provide valuable support for healthcare management such as a medication plan or physiotherapy training. Nevertheless, the current healthcare models typically involve clinical evaluations performed very sparsely (e.g. every 4 to 6 months) because of the physicians’ time and economic constraints. Thus, we are developing robust control-oriented modelling and estimation methods for closely monitoring changes in the patient's state and neurological deficits. These include employing various forms of treatment together with a low-cost wearable sensor (e.g., a smart watch).

Research question 

  • Quantitative characterisation of clinical symptoms and interaction effects of various factors such as medication and fitness level in unconstrained (i.e. daily) environments.
  • Robust modelling and predicting of clinical state of patients against noisy measurements of movements in low-cost sensors.
  • Transferring the clinical insights into control applications for optimising treatment efficacy.

Approach

We are currently developing a machine learning technique for reliably estimating progression and fluctuation of motor deficits in patients with PD using inertia measurements available from a low-cost wearable sensor (e.g., a smart watch). In particular, we employ Gaussian Processes (GP) in order to ensure that the errors from low-cost devices are transparently and robustly processed by the model for estimating a current PD state. 

Key results and achievements 

  • Extraction of tremor information from inertia measurements during daily activities using continuous wavelet transform and machine learning.
  • Parameterisation of PD-specific motions and application of GP for severity estimation

Selected publications

  • M. Lang; F.J. Pfister; J. Fröhner; K. Abedinpour; D. Pichler; U. Fietzek; T.T. Um; D. Kulić; S. Endo; S. Hirche: A Multi-layer Gaussian Process for Motor Symptom Estimation in People with Parkinson’s Disease. IEEE Transactions on Biomedical Engineering, 2019 mehr… BibTeX
  • S. Endo; F.J. Pfister; J. Fröhner; U. Fietzek; D. Pichler; K. Abedinpour; T.T. Um; D. Kulić; M. Lang; S. Hirche: Dynamics-based estimation of Parkinson's disease severity using Gaussian Processes. Second IFAC Conference on Cyber-Physical & Human Systems, 2018 mehr… BibTeX
  • T. Um; F. Pfister; D. Pichler; S. Endo; M. Lang; S. Hirche; U. Fietzek ; D. Kulić: Data augmentation of wearable sensor data for parkinson's disease monitoring using convolutional neural networks. Proceedings of the 19th ACM International Conference on Multimodal Interaction - ICMI 2017, 2017 mehr… BibTeX
  • F. Pfister; D. Kulić; T. Um; D. Pichler; A. Ahmadi; M. Lang; G. König; F. Achilles; S. Endo; K. Abedinpour; K. Ziegler; K. Bötzel; S. Hirche; A. Ceballos-Baumann; U. Fietzek: Deep Learning in Objective Classification of Spontaneous Movement of Patients with Parkinson’s Disease Using Large-Scale Free-Living Sensor Data. International Parkinson and Movement Disorder Society 201721st International Congress, Vancouver, BC mehr… BibTeX
  • P. Beckerle; G. Salvietti; R. Unal; D. Prattichizzo; S. Rossi; G. Castellini; S. Hirche; S. Endo; H.B. Amor; M. Ciocarlie; F. Mastrogiovanni; A.D.Brenna; M. Bianchi: A Human–Robot Interaction Perspective on Assistive and Rehabilitation Robotics. Frontiers in Neurorobotics (11), 2017 mehr… BibTeX

Neuroscientific models of Human-Human and Human-Robot Interaction

Researcher: Satoshi Endo

Motivation

Our research interest centres around neuroscientific understandings of human movement control in dynamic environments, with a special emphasis on motor coordination with another agent including a robotic system. We conduct scientific research about how neuroscientific variables such as perception, cognition, and motor control influence the interaction dynamics of multi-agent coordination. The results are used to derive techniques for effectively controlling mult-agent interactions in, for example, assistive technologies.  

Research questions

  • Perceptual and cognitive factors attributing observation of external input as intentional or erroneous.
  • Evolution of cooperative strategies between agents
  • Models of intention recognition based on interaction dynamics and perceptual cues

Approach

  • Our method consists of modelling human behavior as a stochastic dynamical system and studying how the coordination is moderated by human agents. Human-driven models are then implemented for adaptive control of a robotic system in HRI tasks.

Key results and achievements

  • Modelling multi-agent coordination as a dynamical system to statistically characterise the coordination strategy.
  • Application of bio-inspired motions and human models for object manipulation in pHRI. 

Selected publications

  • P. Donner; S. Endo; M. Buss: Physically Plausible Wrench Decomposition for Multieffector Object Manipulation. Transaction on Robotics (34(4)), 2018, 1053-1067 mehr… BibTeX
  • P. Beckerle; G. Salvietti; R. Unal; D. Prattichizzo; S. Rossi; G. Castellini; S. Hirche; S. Endo; H.B. Amor; M. Ciocarlie; F. Mastrogiovanni; A.D.Brenna; M. Bianchi: A Human–Robot Interaction Perspective on Assistive and Rehabilitation Robotics. Frontiers in Neurorobotics (11), 2017 mehr… BibTeX
  • J. R. Medina; S. Endo; S. Hirche: Impedance-based Gaussian Processes for Predicting Human Behavior during Physical Interaction. International Conference on Robotics and Automation 2016, 2016 mehr… BibTeX
  • A.M. Wing; S. Endo; A. Bradbury; D. Vorberg: Optimal feedback correction in string quartet synchronization. Journal of The Royal Society Interface 11 (93), 2014 mehr… BibTeX
  • A. Bartolo; Y. Coello; M.G. Edwards; S. Delepoulle; S. Endo; A.M. Wing.: Contribution of the motor system to the perception of reachable space: an fMRI study. European Journal of Neuroscience, 2014, 1-11 mehr… BibTeX
  • M. Prada; A. Remazeilles; A. Koene; S. Endo: Implementation and experimental validation of Dynamic Movement Primitives for object handover. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2014 mehr… BibTeX

Estimation of human arm impedance for motor behavior models

Researcher: Hendrik Börner, Satoshi Endo

Motivation

Whenever humans execute a desired motor task, the produced joint torques consist of a feedforward and a feedback component. According to a theory of sensorimotor control, the feedforward component is determined a priori using internal models. During execution, deviations from the planned behavior are compensated by the feedback component, which consists of effects of muscle intrinsic viscoelastic properties, short- and long-latency reflexes, and voluntary feedback at cognitive level. As voluntary feedback possesses the longest delays, it may not be sufficient for the maintenance of stability. Thus, the central nervous system must rely on the intrinsic and reflexive feedback components, which can be modeled by a mechanical impedance consisting of inertia, damping, and stiffness. While the inertia is solely determined by limb kinematics, the latter two can be influenced by muscle activation, which enables the a priori cognitive modulation of the respective characteristics. In addition to fundamental insights into motor control, the analysis of such modulation strategies may yield models that are transferable to robotics domains such as human robot collaboration and variable impedance control.

Research questions

  • Estimation of impedance characteristics in multi-joint arm movements
  • Analysis and data-driven modeling of impedance modulation strategies
  • Transfer of models to robotics domains, e.g., human robot interaction

Approach

We have designed an experimental framework specifically tailored to meet the requirements of impedance estimation during multi-joint arm movements. The experimental apparatus is capable of inducing short, precise disturbances during otherwise free motion of the participants. As we intend to analyze transient impedance modulation strategies, we have developed a single trial feedback component isolation method. The obtained impedance characteristics are included in stochastic dynamical systems that model human motor behavior.

Key results and achievements

  • Design of a framework for multi-joint arm impedance estimation
  • Development of a single trial feedback component isolation method

Selected publications

  • J. R. Medina; H. Börner; S. Endo; S. Hirche: Impedance-based Gaussian Processes for Modeling Human Motor Behavior in Physical and Non-physical Interaction. IEEE Transactions on Biomedical Engineering 66 (9), 2019, 2499-2511 mehr… BibTeX
  • H. Börner; S. Endo; S. Hirche: Estimation of Involuntary Impedance in Multi-joint Arm Movements. IFAC Conference on Cyber-Physical & Human Systems (CPHS), 2018 mehr… BibTeX

Embodiment under autonomous control

Researcher: Jakob Fröhner, Satoshi Endo

Motivation 

Various forms of assistive control have been introduced to help users in operating machines as found in smart vehicles and prosthetics for example. Commonly, semi-autonomous controllers are designed to maximise the task performance to compensate for suboptimal inputs by humans, for example, due to a lack of expertise or task knowledge. However, from a user perspective, introducing semi-autonomous control directly reduces controllability of the system, and a controller which maximises the performance may not necessarily bring positive user experience. This can lead to an undesirable outcome in long-term interaction. Therefore, we research and evaluate control designs from user perspectives by studying how well the autonomous system is embodied in the "control system" of the user by adopting psychological tools and theories.

Research questions

  • Evaluating the quality of human-in-the-loop control using the psychological construct of embodiment in a haptic human-machine interaction tasks
  • Human behaviour modelling and characterisation
  • Assessment of haptic feedbacks

Approach

We have designed a haptic human-machine interaction task to evaluate the quality of human-in-the-loop control using a psychological construct of embodiment. For this task, we composed an experimental framework consisting of a haptic assistance and a virtual environment broadcasted through virtual reality goggles. With this framework, we tested the perceived embodiment under different assistive control algorithms in a reaching task with human participants.

Key results and achievements

  • Design of an experimental setup consisting of a 2-DoF haptic device and a virtual environment for human studies.
  • Investigation of subjective embodiment using psychometric scales.

Video: Measuring the quality of human-robot interaction from an user-centric perspective

Selected publications

  • J. Fröhner; P. Beckerle; S. Endo; S. Hirche: An embodiment paradigm in evaluation of human-in-the-loop control. IFAC Conference on Cyber-Physical & Human Systems (CPHS), 2018 mehr… BibTeX

Shared control for human-robot team interaction

Researcher: Selma Music

Motivation

Through advancements in autonomous capabilities, robots have the potential to become humans’ collaborative partners. Robot teams, consisting e.g. of a mobile platform with manipulators, can physically collaborate with humans as well as extend the human reach to dangerous or inaccessible areas. By leveraging the complementary capabilities of humans, who are able to reason and plan in unstructured environments, and robots, which are able to perform repetitive tasks precisely, performance on cooperative tasks such as collaborative manufacturing, search and rescue, logistics, service, and rehabilitation can be enhanced.

Research questions

  • Combine human and robot team decision making and task execution capabilities
  • Share control between the team members
  • Develop novel interaction paradigms through wearable haptic devices

Approach

We propose a novel control approach which decouples the task into subtasks or elementary constructs of a task. The dynamical behavior of the complete interactive system is projected onto the subtask manifolds. The desired control behaviors of subtask dynamics are then achieved on the reduced manifolds. The noninteraction property between subtasks enables the team members to influence only the subtasks which are assigned to them. The proposed approach does not depend on the number of robots within the team nor on the number of human operators which interact with the robot system.

Key results and achievements

  • Design of a shared control architecture which allows the execution of multiple subtasks
  • Integration of wearable haptic devices with suitable mappings into the shared control architecture
  • Application of the approach in a teleoperation scenario for cooperative manipulation tasks

Selected publications

  • S. Music; S. Hirche: Control Sharing in Human-Robot Team Interaction. Annual Reviews in Control (44), 2017, 342-354 mehr… BibTeX
  • M. Angerer; S. Music; S. Hirche: Port-Hamiltonian Based Control for Human-Robot Team Interaction. Proceedings of 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017 mehr… BibTeX
  • S. Music; G. Salvietti; P. Budde gen. Dohmann; F. Chinello; D. Prattichizzo; S. Hirche: Robot Team Teleoperation for Cooperative Manipulation using Wearable Haptics. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017 mehr… BibTeX
  • S. Music; S. Hirche: Networked human-robot team Interaction. International Symposium on Networked Cyber-Physical Systems (NetCPS) 2016 mehr… BibTeX
  • S. Music; S. Hirche: Classification of Human-Robot Team Interaction Paradigms. 1st IFAC Conference on Cyber-Physical & Human-Systems (CPHS), 2016 mehr… BibTeX
  • D. Sieber; S. Music; S. Hirche: Multi-robot manipulation controlled by a human with haptic feedback. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2015 mehr… BibTeX