unsupervised learning for tactile manipulation

When using high dimensional tactile data for in-hand manipulation we need unsupervised algorithms to reduce its dimensions. Not only makes the dimensionality reduction it easier to use tactile information in real-time robot controllers. The overall system also becomes invariant to the specific tactile sensor technology. We use Variational Auto-Encoders for performing this dimensionality reduction. This algorithm performs efficient inference for directed probabilistic modelling where we are interested in the latent representations of tactile data.

In this context, we have transferred this method to recurrent networks, resulting in stochastic recurrent networks. These models can take the sequential nature of tactile data into account.



Processing of tactile sensor data from an artificial skin

This project targets the separation of nonlinearly interacting causes in tactile data obtained from a skin-sensorized robotic gripper. With x being finger positions and o the properties of a hand-held object, let f(x) be the readout of the tactile skin due to finger egomotion and let g(x,o) be the readout from the tactile skin when a particular object o is hand-held.  g and f are identical when there is no object in the hand. We are interested in obtaining both the finger position x and object properties o given data from the readout g.

The deformation of the polymer employed in the artificial skin is not purely elastic, but a combination of elasticity and plasticity. Thus the deformation and hence measured capacity does not only depend on the current pressure but on its history, i.e. the system shows hysteric behaviour. To accurately predict the true pressure from the measured capacity hysteresis compensation must be performed.

We use Gaussian Processes to learn a dynamic model of the artificial skin from force/capacitance time series. We developed an inference algorithm based on probabilistic inference in Markov chains that uses the learnt model to infer the true forces from the capacitance time series.

The separation into intrinsic (caused by skin deformation) and extrinsic (caused by object contact) tactile data is a non-linear blind signal separation problem and we will tackle it using machine learning methods such as independent component analysis (ICA), multimodal autoencoders and restricted Boltzmann machines (RBMs).



anomaly detection

The standard approach to describing system behaviour over time is by devising intricate dynamics models—typically in terms of first- or second-order differential equations—which are based on detailed dynamic models of the system. Such models can then help to both control as well as predict the behaviour of the system, and thus serve as a basis to detect faults.

We investigate the common case where either models are difficult to obtain or are not rich enough to describe the dynamic behaviour of the system. This can happen when the system has many degrees of freedom (DoF), high-dimensional sensors, etc. This is typically true for robotic systems, vehicles, manufacturing sites, etc.—any modern actor–sensor system that we depend on.

In such cases, the quality fault detection deteriorates: too many false positives make the fault detection useless, while too many true negatives may harm the system. Rather than trusting such incomplete models, we rely on a methodology which creates a probabilistic model of the system from data recorded from it, and detect outliers with respect to this learned model. Following standard procedures, detecting such outliers we call anomaly detection.

This type of detection is notoriously difficult as it is an ill-posed problem. First, the notion of anomaly hugely depends on the domain. Then, the boundary between “normal” and “anomalous” might not be precise and might evolve over time. Anomalies might appear normal or be obscured by noise. Finally, collecting anomalous data is very difficult, and labelling them even more so. Two observations are important to make: (i) anomalies are sparse by their very nature, and (ii) in a high-dimensional real-world scenario it will not be possible to rigorously define “normal” and “anomalous” regions of the data space. We therefore focus on an unsupervised approach. In a first step a probabilistic model of the system’s data is created. This model is then used to identify (patterns of) samples that do not fit in the probabilistic model, which we then conjecture to correspond to the sought anomalies.


Automatic fault detection in silicon wafers using convolutional neural networks

The production of integrated circuits (ICs) is a complex operation during which silicon wafers are processed in multiple stages, each of which can introduce faults. They can be either spatially isolated and thus affect only a limited number of ICs on the wafer or make the whole wafer unusable. Many faults can be detected optically with the naked eye; a skilled engineer can visually detect most faults by either looking at the wafer directly or examine photographs of it taken from a variety of viewing angles.

The aim of this project is to automatically detect faulty regions in wafer images using a custom computer vision pipeline. The main detection stage consists of a deep, multi-view convolutional neural network (CNN) that classifies patches of the wafer image as either good or bad. To deal with faults at different scales we employ a set of CNNs trained at different resolutions (original resolution and artificially downsampled versions of the wafer image) to find both small defects (dust or scratches) and large-scale defects (inhomogeneous distribution of chemicals).

Picture of  Justin Bayer

Justin Bayer

Picture of  Marvin Ludersdorfer

Marvin Ludersdorfer

fortiss: PhD candidate
anomaly detection
Picture of  Maximilian Karl

Maximilian Karl

TUM: PhD candidate
efficient inference
Picture of  Maximilian Soelch

Maximilian Soelch

TUM: PhD candidate
robot control with deep learning
Picture of  Sebastian Urban

Sebastian Urban

TUM: PhD candidate
learning skin data
surbantumde, +49 89 289-25794


    Sebastian Urban, Marvin Ludersdorfer, Patrick van der Smagt (2015). Sensor Calibration and Hysteresis Compensation with Heteroscedastic Gaussian Processes. IEEE Sensors. 15 (11), 6498--6506.
    Zoltán A. Milacskí, Marvin Ludersdorfer, Patrick van der Smagt, Andräs Lörincz (2015). Robust Detection of Anomalies via Sparse Methods. Proc. ICONIP


    Atzori M, Gijsberts A, Heynen S, Mittaz-Hager A, Deriaz O, Smagt P van der, Castellini C, Caputo B, Müller H (2012). Building the NINAPRO Database: A Resource for the Biorobotics Community. IEEE International Conference on Biomedical Robotics and Biomechatronics 1258--1265.


    Smagt P van der, Grebenstein M, Urbanek H, Fligge N, Strohmayr M, Stillfried G, Parrish J, Gustus A (2009). Robotics of human movements. Journal of physiology, Paris. 103 (3-5), 119-132.


    Arbib M, Metta G, Smagt P van der (2008). Neurorobotics: From Vision to Action. In B. Siciliano and O. Khatib (Eds.) Springer Handbook of Robotics 1453--1480.


    Smagt P van der, Hirzinger G (2006). NEUROBOTICS: The fusion of neuroscience and robotics. Proceedings ISR-2006, Joint Conference on Robotics / ROBOTIK


    Urbanek H, Albu-Schäffer A, Smagt P van der (2004). Learning from demonstration: Repetitive movements for autonomous service robotics. 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 3495-3500.


    Smagt P van der, Bullock D (2002). Guest editorial for Special Issue on Scalable Applications of Neural Networks to Robotics. Applied Intelligence. 17 7--10.


    Omidvar O, Smagt P van der (1997). Neural Systems for Robotics. Academic Press: Boston, Massachusetts