ML for movement modelling
Based on measured data, we create machine-learning based models of kinematic and dynamic properties movement in robots and humans. Our methodologies focus on recurrent neural networks, Dynamic Motion Primitives, Autoencoders, and deep learning.
Apart from these models, we also create biomechanical models of hands and arms, based on measurement of the human movement system.
movement in latent space
In the standard formulation, dynamic movement primitives (DMPs) suffer from suboptimal generalisation—when used in configuration space—or are a victim of the curse of dimensionality—when used in task space. To solve this problem we propose a model called autoencoded dynamic movement primitive (AEDMP) which uses deep autoencoders to find a representation of movement in a latent feature space, in which DMP can optimally generalise. The architecture embeds DMP into such an autoencoder and allows the whole to be trained as a unit.
The objectives are:
- A major strength of our method is caused by integrating the dimensionality reduction with the DMP-based movement representation.
- Generate new movements which are not in the training data set by simply switching on/off or interpolating one hidden unit.
- The model facilitates the reconstruction of missing joints and missing sub sequences.
The figure shows movement encoded in two hidden neurons (the value of each on the vertical viz. horizontal axis).
grip force measurement
Estimating human fingertip forces is required to understand force distribution in grasping and manipulation.
Human grasping behaviour can then be used to develop force and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this project we develop approaches which uses images of the human fingertip to reconstruct grip force and torque at the finger.
The approaches include the following steps:
- Image alignment: develop a method for 2d image to 3d finger model alignment method using convolutional neural networks (CNNs); employ non-rigid image registration method.
- Force predictor: predict the forces from fingertip images using Gaussian process or neural networks.
The objectives are:
- Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable.
- Moving away from a constrained lab setting (e.g., finger brace) with perfect conditions and comfortable restrictions, we deal with an additional constraint that strongly influences measurement results: the bending of the finger. With the (optically measured) angles of the finger joints, we use this as extra parameter for our estimator, leading to much improved accuracies when predicting the finger-exerted force from the nail image.
Another major constraint setting is the contract surface. Our approach is robust for various contact surfaces and furthermore it can predict the types of the contact surface.
Kinematics of the human hand
The amazing manipulation capabilities that we develop show us clearly the versatility of the human hand. But even everyday tasks like picking a coin from a wallet are---from a robotics point of view---utterly impressive. What is so special about our hands?
|Pose estimation of a bone. The points are extracted from the MRI images. The bone shown in blue on the left is taken from one MRI image and the one in red on the right from another. The pose estimation algorithm determines the movement that is necessary to match the blue and red points. The blue points on the right show the result of the pose estimation.|
In cooperation with Rechts der Isar hospital, Munich, we took a large series (~50 images) of magnetic resonance images (MRI) of a healthy human hand in different postures. MRI allows three-dimensional views of the inside of the human body. The method works by measuring the response of hydrogen atoms inside the body to magnetic stimulation and is – unlike CT imaging – non-ionising.
To derive a kinematic model from the MRI images, we conducted the following steps:
- Segmentation: Highlight the data that belongs to each individual bone.
- Pose estimation: Determine the position and orientation ("pose") of each bone with respect to a reference pose.
- Identification of joint axes: Numerically determine the position and orientation of joint axes that optimally incorporate the measured bone poses, using different joint models (one or two axes, non-intersecting or intersecting axes).
- Build the hand model: Select joint types that appropriately fulfill the compromise between accuracy and complexity, and combine the joints to so-called kinematic chains.
|Resulting hand model with 24 degrees of freedom. The index finger metacarpal bone (marked by a black square) is taken as the base of the model. In joints with two axes, the first axis is shown red and the second one green.|
For the pose estimation we used an algorithm that the robot Justin uses to identify the location of objects on a table. (The task is similar: Matching three-dimensional point clouds.)
The resulting hand model is shown to the right. The base of the model is the index finger metacarpal bone ("palm bone"), marked by a black square. From there, the kinematic chains extend, indicated by black lines. A kinematic chain is a series of joints, where the position of the last link (in this case the fingertip) depends on the joint angles of all joints in the chain.
The first joint of the thumb is modeled by two non-intersecting axes of rotation, connected by a thick line. The second joint of the thumb also exhibits significant side ward movement and is therefore also modeled by two joint axes, in this case intersecting ones.
The four fingers all have one axis of rotation that allows for a side ward movement and three axes for bending and stretching. The arching of the palm takes place around three axes pointing roughly in the direction of the long axes of the palm bones.
Apart from kinematics, other aspects of the human hand are also important for its fine manipulation abilities, for example touch sensing, motion planning and motion control.
Impedance of the human arm
Defining the Cartesian stiffness matrix of variable-impedance robots is a quite heuristic task. Furthermore, depending on the desired task the stiffness behaviour must be adapted during movement. Humans learn to control limb stiffness from interaction, and we indeed exhibit fine variation of impedance depending on the task and environment. But how? We want to understand the mechanisms for setting and varying impedance in the human arm and hand, and transfer such models to the robotic domain.
Our main goals are
- to understand according to which cost functions biological systems adjust their impedance, and how does intrinsic---defined by the skeletomuscular structure---impedance play a role or, conversely, how and why does the nervous system fluctuate impedance;
- use the gained knowledge to improve body-machine interfaces and to pave the way towards modern impedance teleoperated systems (including prosthetic devices, rehabilitation devices, tele-surgical robotic systems, and so on).
We have developed different impedance measurement methods for identifying impedance of the human fingers, arms, and legs. We combine classical perturbation approaches with EMG-based identification, using force-torque-sensors and optical tracking systems.