We focus on machine learning, mostly deep learning and recurrent neural networks, and its application to movement representation, robotics, and sensor integration.

We use and further develop deep learning, deep recurrent neural networks, convolutional neural networks, autoencoders, etc. and use them in various data related to movement, robotics, and sensory processing.

Using deep learning, we do end-to-end learning on visual and tactile data.  We also develop methods for anomaly detection.

Using our machine-learning methodologies, we model movement in latent domains and use those in movement prediction, analysis, and control.  These models are combined with biomechanical and motor control models.

Limb rehabilitation and assistive robotics are paramount applications of the techniques developed in biomimetic robotics. Using our machine-learning methods, we focus upon human-computer interfaces to aid the disabled regain the lost limb functionality. In our view, both rehabilitation and prosthetics rely on re-establishing the sensori-motor loop with the missing limb.



We'd like you to devise and consolidate a system which robustly predicts robot and human arm movement based on visual data.  You will be working with variational autoencoders, recurrent neural networks, and convolutional neural networks in a lab which focuses on machine learning, deep learning, and robotics.

You are asked to work on real robotic data to understand control parameters of a light-weight robotic arm. 

Why are robotic hands so clumsy?  In an attempt to solve this problem, we investigate the use of tactile sensors for robotic in-hand manipulation. In particular, based on a number of tactile sensors (the BioTac; the DLR tactile sensor; and the iCub tactile sensor), we want to create algorithms that can use such sensing to manipulate objects. In cleartext: write, unscrew a bulb, or hand over a knife with a robotic hand.

In 2016, Patrick and his lab moved to Volkswagen Group in Munich.  They continue doing fundamental research on machine learning and control.  Their new webspace is at argmax.ai.




Your name could be here

want to join our team? check out the positions on the left.



Below our 15 most recent publications. If you need more, follow the link. And note: All downloadable PDFs are for personal use only. Please do not redistribute.

Markus Kühne, Johannes Potzy, R Garcia-Rochin, Patrick van der Smagt, Angelika Peer (2017). Design and Evaluation of a Haptic Interface With Octopod Kinematics. IEEE/ASME Transactions on Mechatronics. 22 (5), 2091-2101.
Benedikt Staffler, Manuel Berning, Kevin M Boergens, Anjali Gour, Patrick van der Smagt, Moritz Helmstaedter (2017). SynEM, automated synapse detection for connectomics. eLife. 2017;6:e26414
Sebastian Urban, Patrick van der Smagt (2017). Automatic Differentiation for Tensor Algebras. arXiv.
Sebastian Urban, Marcus Basalla, Patrick van der Smagt (2017). Gaussian Process Neurons Learn Stochastic Activation Functions. arXiv.
Baris Kayalibay, Grady Jensen, Patrick van der Smagt (2017). CNN-based Segmentation of Medical Imaging Data. arXiv.
Joern Vogel, N Takemura, Hannes Höppner, Patrick van der Smagt, Gowrishankar Ganesh (2017). Hitting the sweet spot: Automatic optimization of energy transfer during tool-held hits. IEEE International Conference on Robotics and Automation (ICRA) 1549-1556 .
Rui Zhao, Ali Haider, Patrick van der Smagt (2017). Two-Stream RNN/CNN for Action Recognition in 3D Videos. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017)
Maximilian Karl, Maximilian Soelch, Justin Bayer, Patrick van der Smagt (2017). Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data. International Conference on Learning Representations (ICLR)
Holger Urbanek, Patrick van der Smagt (2016). iEMG: Imaging Electromyography. Journal of Electromyography and Kinesiology. 27 1--9.
Agneta Gustus, Patrick van der Smagt (2016). Evaluation of joint type modelling in the human hand. Journal of Biomechanics. 49 (13), 3097–3100.
Christopher Wolf, Maximilian Karl, Patrick van der Smagt (2016). Variational Inference with Hamiltonian Monte Carlo. arXiv.
Maximilian Karl, Maximilian Soelch, Justin Bayer, Patrick van der Smagt (2016). Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data. arxiv.
Nutan Chen, Maximilian Karl, Patrick van der Smagt (2016). Dynamic Movement Primitives in Latent Space of Time-Dependent Variational Autoencoders. Proc. 16th IEEE-RAS International Conference on Humanoid Robots
Herke van Hoof, Nutan Chen, Maximilian Karl, Patrick van der Smagt, Jan Peters (2016). Stable Reinforcement Learning with Autoencoders for Tactile and Visual Data. Proc. IEEE International Conference on Intelligent Robots and Systems (IROS)
Wiebke Koepp, Patrick van der Smagt, Sebastian Urban (2016). A Differentiable Transition Between Additive and Multiplicative Neurons. International Conference on Learning Representations (ICLR)