machine learning

Natural data as found in biological signals or images is usually highly redundant and noisy. Classical models for the stochasticity in such processes break down in many such cases. For example, due to the presence of edges in images, the gradients are giving rise to fat tailed distributions. On the other hand, it can easily be seen that multiple EMG signals are highly non-Gaussian.

In machine learning, we investigate methods for finding useful representations of natural data. For this, we use non-linear parametric models. These are combined into deep and recurrent architectures which are subsequently optimised with classical and novel optimisation techniques on a wide variety of objectives.

The objectives typically encourage the representations to fulfil some numerical criterion: sparsity, independence, clustering of similar items or the ability to reconstruct the input. The models we use include but are not limited to deep belief networks, recurrent neural networks, convolutional neural networks, variational autoencoders, and Gaussian Processes.

 

 

Fast Adaptive Weight Noise

We developed an efficient calculation of the marginal likelihood of a distribution over the weights for neural networks. We use a technique called Variance Propagation for computing mean and variance when propagating a Gaussian distribution through a neural network. (Wang & Manning 2013) are providing rules for propagation mean and variance through a set of linear transformations and nonlinear transfer functions. By choosing Gaussian distributions for the network weights and propagating this uncertainty through the network we can efficiently calculate the marginal likelihood. Optimising it directly with respect to the parameters of the distribution of the weights will lead to a maximum likelihood approach. By adding a KL-divergence between the distribution over the weights and a prior we prevent the model from overfitting the data.

A slight variant of this is to use variance propagation to approximate Bayesian learning of neural networks: we can optimise the variational upper bound on the negative log-likelihood of the data. This allows to exploit model uncertainty in a wide range of scenarios, such as active learning or reinforcement learning. Apart from being able to model uncertainty it also requires very few data.

 

 

Hybrid addition-multiplication networks using parameterisable transfer functions

Can the performance of neural networks be improved by the use of a novel, parameterizable transfer function that allows each neuron to smoothly adjust the operation it performs on its inputs between summation and multiplication?

In artificial neural networks the value of a neuron is given by a weighted sum of its inputs propagated through a non-linear transfer function; however some tasks greatly benefit from units that compute the product instead of the sum of their inputs.

To allow neurons to autonomously determine whether they are additive or multiplicative, we propose a parameterisable transfer function based on the fractionally iterated exponential function generated from a solution to Schröder’s functional equation. This class of transfer functions allows to continuously interpolate the operation a neuron performs between addition and multiplication. Since it is also differentiable, the operation can be determined using standard backpropagation training for neural networks.

So far the mathematical theory has been established (Urban & van der Smagt, 2015) and an implementation effort is currently being made. Next steps will include testing of this novel transfer function on regression networks.

Picture of  Daniela Korhammer

Daniela Korhammer

alumni
Picture of  Justin Bayer

Justin Bayer

alumni
bayersensedio
Picture of  Maximilian Karl

Maximilian Karl

TUM: PhD candidate
efficient inference
Picture of  Nutan Chen

Nutan Chen

TUM: PhD candidate
hand modelling
nutanin.tumde
Picture of  Patrick van der Smagt

Patrick van der Smagt

current: Head of AI Research, data lab, VW Group

Previous: Director of BRML labs
fortiss, An-Institut der Technischen Universität München
Professor for Biomimetic Robotics and Machine Learning, TUM

Chairman of Assistenzrobotik e.V.
smagtbrmlorg
Picture of  Sebastian Urban

Sebastian Urban

TUM: PhD candidate
learning skin data
surbantumde, +49 89 289-25794



2017

    Benedikt Staffler, Manuel Berning, Kevin M Boergens, Anjali Gour, Patrick van der Smagt, Moritz Helmstaedter (2017). SynEM, automated synapse detection for connectomics. eLife. 2017;6:e26414
    Baris Kayalibay, Grady Jensen, Patrick van der Smagt (2017). CNN-based Segmentation of Medical Imaging Data. arXiv.
    Rui Zhao, Ali Haider, Patrick van der Smagt (2017). Two-Stream RNN/CNN for Action Recognition in 3D Videos. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017)
    Maximilian Karl, Maximilian Soelch, Justin Bayer, Patrick van der Smagt (2017). Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data. International Conference on Learning Representations (ICLR)

2016

    Christopher Wolf, Maximilian Karl, Patrick van der Smagt (2016). Variational Inference with Hamiltonian Monte Carlo. arXiv.
    Maximilian Karl, Maximilian Soelch, Justin Bayer, Patrick van der Smagt (2016). Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data. arxiv.
    The Theano Development Team (2016). Theano: A Python framework for fast computation of mathematical expressions. arxiv.
    Nutan Chen, Maximilian Karl, Patrick van der Smagt (2016). Dynamic Movement Primitives in Latent Space of Time-Dependent Variational Autoencoders. Proc. 16th IEEE-RAS International Conference on Humanoid Robots
    Herke van Hoof, Nutan Chen, Maximilian Karl, Patrick van der Smagt, Jan Peters (2016). Stable Reinforcement Learning with Autoencoders for Tactile and Visual Data. Proc. IEEE International Conference on Intelligent Robots and Systems (IROS)
    Wiebke Koepp, Patrick van der Smagt, Sebastian Urban (2016). A Differentiable Transition Between Additive and Multiplicative Neurons. International Conference on Learning Representations (ICLR)
    Maximilian Sölch, Justin Bayer, Marvin Ludersdorfer, Patrick van der Smagt (2016). Variational Inference for On-line Anomaly Detection in High-Dimensional Time Series. International Conference on Learning Representations (ICLR)

2015

    Sebastian Urban, Marvin Ludersdorfer, Patrick van der Smagt (2015). Sensor Calibration and Hysteresis Compensation with Heteroscedastic Gaussian Processes. IEEE Sensors. 15 (11), 6498--6506.
    Maximilian Karl, Justin Bayer, Patrick van der Smagt (2015). Efficient Empowerment.
    Nutan Chen, Justin Bayer, Sebastian Urban, Patrick van der Smagt (2015). Efficient movement representation by embedding Dynamic Movement Primitives in Deep Autoencoders. Proc. 2015 IEEE-RAS International Conference on Humanoid Robots
    Justin Bayer, Maximilian Karl, Daniela Korhammer, Patrick van der Smagt (2015). Fast Adaptive Weight Noise. arXiv
    Zoltán A. Milacskí, Marvin Ludersdorfer, Patrick van der Smagt, Andräs Lörincz (2015). Robust Detection of Anomalies via Sparse Methods. Proc. ICONIP
    Sebastian Urban, Patrick van der Smagt (2015). A Neural Transfer Function for a Smooth and Differentiable Transition Between Additive and Multiplicative Interactions. arXiv
    P. Fischer, A. Dosovitskiy, E Ilg, P Haeusser, C Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, T. Brox (2015). FlowNet: Learning Optical Flow with Convolutional Networks. Proc. ICCV

2014

    Justin Bayer, Christian Osendorfer (2014). Learning Stochastic Recurrent Networks. Workshop on Advances in Variational Inference, Neural Information Processing Systems 2014
    Hornung R, Urbanek H, Klodmann J, Osendorfer C, Smagt P van der (2014). Model-free robot anomaly detection. Proc. IEEE/RSJ International Conference on Robotics and Systems (IROS)
    Justin Bayer, Christian Osendorfer, Daniela Korhammer, Nutan Chen, Sebastian Urban, Patrick van der Smagt (2014). On Fast Dropout and its Applicability to Recurrent Networks. Proceedings of the International Conference on Learning Representations
    Christian Osendorfer, Hubert Soyer, Patrick van der Smagt (2014). Image Super-Resolution with Fast Approximate Convolutional Sparse Coding. Neural Information Processing 8836
    Nutan Chen, Sebastian Urban, Christian Osendorfer, Justin Bayer, Patrick van der Smagt (2014). Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian Processes. Robotics and Automation (ICRA), 2014 IEEE International Conference on

2013

    Rückstieß T, Osendorfer C, Smagt P van der (2013). Minimizing Data Consumption with Sequential Online Feature Selection. International Journal of Machine Learning and Cybernetics. 4 (3), 235-243.
    Sebastian Urban, Justin Bayer, Christian Osendorfer, Göran Wesling, Benoni B. Edin, Patrick van der Smagt (2013). Computing grip force and torque from finger nail images using Gaussian processes. Proc. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 4034--4039.
    Justin Bayer, Christian Osendorfer, Sebastian Urban, Patrick van der Smagt (2013). Training Neural Networks with Implicit Variance. Neural Information Processing 8227 132-139.
    Christian Osendorfer, Justin Bayer, Sebastian Urban, Patrick van der Smagt (2013). Convolutional Neural Networks learn compact local image descriptors. Neural Information Processing 8228 624-630.

2012

    Bayer J, Osendorfer C, Smagt P van der (2012). Learning sequence neighbourhood metrics. In Villa, Alessandro and Duch, Włodzisław and \'Erdi, Péter and Masulli, Francesco and Palm, Günther (Eds.) Artificial Neural Networks and Machine Learning – ICANN 2012 7552 531-538.

2011

    Rückstiess T, Osendorfer C, Smagt P van der (2011). Sequential feature selection for classification. In Dianhui Wang and Mark Reynolds (Eds.) AI 2011: Advances in Artificial Intelligence 7106
    Osendorfer C, Schlüter J, Schmidhuber J, Smagt P van der (2011). Unsupervised learning of low-level audio features for music similarity estimation. Workshop on Learning Architectures, Representations, and Optimization for Speech and Visual Information Processing, ICML 2011

1998

    Smagt P van der, Hirzinger G (1998). Solving the Ill-Conditioning in Neural Network Learning. In J. Orr and K. Müller (Eds.) Neural Networks: Tricks of the Trade 193--206.
    Smagt P van der, Hirzinger G (1998). Why Feed-Forward Networks are in a Bad Shape. In L. Niklasson and M. Bodén and T. Ziemke (Eds.) Proceedings of the 8th International Conference on Artificial Neural Networks Springer: 159--164.

1995

    Smagt P van der, Kröse B (1995). Using Many-Particle Decomposition to get a Parallel Self-Organising Map. In J. van Vliet (Eds.) Proceedings of the 1995 Conference on Computer Science in the Netherlands 241-249.
    Smagt P van der, Groen F (1995). Approximation with neural networks: Between local and global approximation. Proceedings of the 1995 International Conference on Neural Networks II:1060-II:1064.
    Smagt P van der (1995). Visual Robot Arm Guidance using Neural Networks. Ph.D. thesis: Dept of Computer Systems, University of Amsterdam
    Jansen A, Smagt P van der, Groen F (1995). Nested Networks for Robot Control. In A. F. Murray (Eds.) Applications of Neural Networks 221--239.

1994

    Smagt P van der (1994). Minimisation methods for training feed-forward networks. Neural Networks. 7 (1), 1--11.
    Hesselroth T, Sarkar K, Smagt P van der, Schulten K (1994). Neural network control of a pneumatic robot arm. IEEE Transactions on Systems, Man, and Cybernetics. 24 (1), 28--38.
    Smagt P van der, Groen F, Groenewoud F van het (1994). The locally linear nested network for robot manipulation. Proceedings of the IEEE International Conference on Neural Networks 2787--2792.
    Smagt P van der, Groen F, Groenewoud F van het (1994). Robotic Hand-Eye Coordination using Multi-Resolution Linear Perceptron Representation. In H. M. Groenboom and H. W. Klijn Hesselink and M. M. Lankhorst (Eds.) Proceedings of the 1994 Groningen Student Conference on Computer Science 85--92.
    Kröse B, Smagt P van der (1994). An Introduction to Neural Networks. University of Amsterdam: Amsterdam, The Netherlands

1993

    Smagt P van der, Schulten K (1993). Control of pneumatic robot arm dynamics by a neural network. World congress on neural networks III/180-III/183.
    Kröse B, Smagt P van der, Groen F (1993). A one-eyed self-learning robot manipulator. In G. Bekey and K. Goldberg (Eds.) Neural networks in robotics 19-28.
    Groen F, Kröse B, Smagt P van der, Bartholomeus MGP, Noest AJ (1993). Neural Networks for robot eye-hand coordination. In S. Gielen and B. Kappen (Eds.) Artificial neural networks 211-218.

1992

    Smagt P van der, Jansen A, Groen F (1992). Interpolative robot control with the nested network approach. IEEE Int. Symposium on Intelligent Control 475-480.
    Jansen A, Smagt P van der, Groen F (1992). High-precision robot control: The nested network. In I. Aleksander and J. Taylor (Eds.) Artificial Neural Networks 2 583-586.

1991

    Groen F, Kröse B, Smagt P van der (1991). Parallel Distributed Processing in Autonomous Robot Systems. Proceedings of the 1991 Symposium on Neural Networks 24--25.

1990

    Smagt P van der (1990). A Comparative Study of Neural Network Algorithms Applied to Optical Character Recognition. Proceedings of the Third International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems 1037--1044.