Research Overview

Welcome to the AIT Lab where AI meets Interactive Technologies. We do research at the intersection of Machine Learning, Computer Vision and Human Computer Interaction (HCI). Our main mission is to develop new ways for humans to interact with complex interactive systems (computers, wearables, robots), powered by advanced algorithms and technologies in machine perception, planning and data driven user modelling. Please see research and publications for more information.

The AIT Lab, led by Prof. Dr. Otmar Hilliges, is within the Department of Computer Science at ETH Zurich. The group is jointly affiliated with the Institute of Visual Computing and Institute of Pervasive Computing.

Latest News

13.10.2021
The two papers on KinectFusion win lasting impact award at UIST (link) and impact award at ISMAR (link).
28.09.2021
Prof. Dr. Otmar Hilliges is appointed as Max Planck Fellow at MPI-IS. Congratulations!
11.08.2021
Zijian Dong joins the AIT lab. Welcome!
01.08.2021
Our former PostDoc Xucong Zhang starts as an Assistant Professor at TU Delft. Congratulations!
23.07.2021
We have 7 papers (1 oral) accepted at ICCV. Stay tuned for more details.
Check here for previous news.

Open PostDoc positions

We are currently looking for 2 post-docs with a background in either i) machine learning, bayesian optimisation, active learning, variational methods, deep-learning or related areas or ii) computer vision, graphics, robotics, numerical methods or physical simulation.

Excellent programming skills in C++ or other high-level language are expected and experience in Matlab, Python, OpenGL, or CUDA are beneficial. Fluent communication skills (written and presentation) in English are essential.

Positions are fully funded (via ERC starting grant) and are available to those of any nationality (some knowledge of German is beneficial but not essential). For more details see how to apply.


Latest Projects


Teaser-Picture

Human Performance Capture from Monocular Video in the Wild


Teaser-Picture

A Skeleton-Driven Neural Occupancy Representation for Articulated Hands

AuthorsK. Karunratanakul, A. Spurr, Z. Fan, O. Hilliges, S. Tang
In ProceedingsInternational Conference on 3D Vision (3DV), 2021

Accepted as Oral Presentation

Teaser-Picture

Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation

AuthorsZ. Fan, A. Spurr, M. Kocabas, S. Tang, M. Black, O. Hilliges
In ProceedingsInternational Conference on 3D Vision (3DV), 2021

Accepted as Oral Presentation

Teaser-Picture

A Spatio-temporal Transformer for 3D Human Motion Prediction


VariTex: Variational Neural Face Textures


Teaser-Picture

EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers

AuthorsM. Kaufmann, Y. Zhao, C. Tang, L. Tao, C. Twigg, J. Song, R. Wang, O. Hilliges
In ProceedingsInternational Conference on Computer Vision (ICCV), 2021

Teaser-Picture

SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes


Teaser-Picture

Shape-aware Multi-Person Pose Estimation from Multi-view Images


Teaser-Picture

Self-Supervised 3D Hand Pose Estimation from monocular RGB via Contrastive Learning

AuthorsA. Spurr*, A. Dahiya*, X. Wang, X. Zhang, O. Hilliges
In ProceedingsInternational Conference on Computer Vision (ICCV), 2021
* These two authors contributed equally to this work.

Accepted as Oral Presentation

Teaser-Picture

SPEC: Seeing People in the wild with an Estimated Camera

AuthorsM. Kocabas, C. Huang, J. Tesch, L. Müller, O. Hilliges, M. Black
In ProceedingsProceedings of the International Conference on Computer Vision 2021, Oct 2021

Teaser-Picture

PARE: Part Attention Regressor for 3D Human Body Estimation


Teaser-Picture

Hedgehog: Handheld Spherical Pin Array based on a CentralElectromagnetic Actuator

AuthorsA. Aline, J. Zarate, T. Langerak, V. Vechev, O. Hilliges
In ProceedingsWorld Haptics Conference, 2021

Honorable Mention - Best Paper Award

Learning Functionally Decomposed Hierarchies for Continuous Control Tasks With Path Planning

AuthorsS. Christen*, L. Jendele*, E. Aksan, O. Hilliges
In IEEE Robotics and Automation Letters (Volume: 6, Issue: 2), 2021
* These two authors contributed equally to this work.

Accepted as Oral Presentation at the Deep RL Workshop at NeurIPS

Teaser-Picture

Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation

AuthorsN. Vulin, S. Christen, S. Stevsic, O. Hilliges
In IEEE Robotics and Automation Letters (Volume: 6, Issue: 2), April 2021

Optimization-based User Support for Cinematographic Quadrotor Camera Target Framing


Teaser-Picture

Hierarchical Reinforcement Learning Explains Task Interleaving Behavior


CoSE: Compositional Stroke Embeddings

AuthorsE. Aksan, T. Deselaers, A. Tagliasacchi, O. Hilliges
In ProceedingsAdvances in Neural Information Processing Systems, 2020

Self-Learning Transformations for Improving Gaze and Head Redirection


Teaser-Picture

Spatial Attention Improves Iterative 6D Object Pose Estimation


Teaser-Picture

Convolutional Autoencoders for Human Motion Infilling


Teaser-Picture

Omni: Volumetric Sensing and Actuation of Passive Magnetic Tools for Dynamic Haptic Feedback

AuthorsT. Langerak*, J. Zarate*, D. Lindlbauer, C. Holz, O. Hilliges
In ProceedingsProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, Virtual Event, USA, 2020
* These two authors contributed equally to this work.

Teaser-Picture

Optimal Control for Electromagnetic Haptic Guidance Systems


Teaser-Picture

Capturing Subjective First-Person View Shots with Drones for Automated Cinematography

AuthorsA. Ashtari, S. Stevsic, T. Nägeli, J. Bazin, O. Hilliges
In ProceedingsACM Transactions on Graphics (Proceedings of ACM SIGGRAPH) (Volume: 39, Issue: 5), Aug 2020

Teaser-Picture

Learning-based Region Selection for End-to-End Gaze Estimation

AuthorsX. Zhang, Y. Sugano, A. Bulling, O. Hilliges
In ProceedingsBritish Machine Vision Virtual Conference (BMVC), 2020

Teaser-Picture

Human Body Model Fitting by Learned Gradient Descent

AuthorsJ. Song*, X. Chen*, O. Hilliges
In ProceedingsEuropean Conference on Computer Vision (ECCV), 2020
* These two authors contributed equally to this work.

Teaser-Picture

Category Level Object Pose Estimation via Neural Analysis-by-Synthesis

AuthorsX. Chen*, Z. Dong*, J. Song, A. Geiger, O. Hilliges
In ProceedingsEuropean Conference on Computer Vision (ECCV), 2020
* These two authors contributed equally to this work.

Teaser-Picture

Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints

AuthorsA. Spurr, U. Iqbal, P. Molchanov, O. Hilliges, J. Kautz
In ProceedingsEuropean Conference on Computer Vision (ECCV), Glasgow, Scotland, 2020

Winner of HANDS2019 challenge: RGB-based 3D Hand Pose Estimation while Interacting with Objects

Teaser-Picture

ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation

AuthorsX. Zhang, S. Park, T. Beeler, D. Bradley, S. Tang, O. Hilliges
In ProceedingsEuropean Conference on Computer Vision (ECCV), 2020

Accepted as Spotlight Presentation

Towards End-to-end Video-based Eye-Tracking


Teaser-Picture

Contact-free Nonplanar Haptics with a Spherical Electromagnet

AuthorsJ. Zarate*, T. Langerak*, B. Thomaszewski, O. Hilliges
In Proceedings2020 IEEE Haptics Symposium (HAPTICS), 2020
* These two authors contributed equally to this work.

Teaser-Picture

Detecting Relevance during Decision-Making from Eye Movements for UI Adaptation

AuthorsA. Feit*, L. Vordemann*, S. Park, C. Bérubé, O. Hilliges
In ProceedingsACM Symposium on Eye Tracking Research and Applications (ETRA), 2020
* These two authors contributed equally to this work.

Teaser-Picture

Accurate Real-time 3D Gaze Tracking Using a Lightweight Eyeball Calibration


Teaser-Picture

Learning to Assemble: Estimating 6D Poses for Robotic Object-Object Manipulation

AuthorsS. Stevsic, S. Christen, O. Hilliges
In ProceedingsIEEE Robotics and Automation Letters (Volume: 5, Issue: 2), April 2020

Teaser-Picture

Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks


Teaser-Picture

Monocular Neural Image Based Rendering with Continuous View Control

AuthorsX. Chen*, J. Song*, O. Hilliges
In ProceedingsInternational Conference on Computer Vision (ICCV), Oct 2019
* These two authors contributed equally to this work.

Teaser-Picture

End-to-end Learning for Graph Decomposition

AuthorsJ. Song, B. Andres, M. Black, O. Hilliges, S. Tang
In ProceedingsInternational Conference on Computer Vision (ICCV), Oct 2019

Few-Shot Adaptive Gaze Estimation

AuthorsS. Park*, S. De Mello*, P. Molchanov, U. Iqbal, O. Hilliges, J. Kautz
In ProceedingsInternational Conference on Computer Vision (ICCV), Seoul, Korea, 2019
* These two authors contributed equally to this work.

Accepted as Oral Presentation

Teaser-Picture

Structured Prediction Helps 3D Human Motion Modelling

AuthorsE. Aksan*, M. Kaufmann*, O. Hilliges
In ProceedingsThe IEEE International Conference on Computer Vision (ICCV), Oct 2019
* These two authors contributed equally to this work.

Content-Consistent Generation of Realistic Eyes with Style

AuthorsM. C. Buehler, S. Park, S. De Mello, X. Zhang, O. Hilliges
In ProceedingsInternational Conference on Computer Vision Workshops (ICCVW), Seoul, Korea, 2019

OpenEDS Image Generation Challenge Winner

Teaser-Picture

Learning Cooperative Personalized Policies from Gaze Data

AuthorsC. Gebhardt, B. Hecox, B. van Opheusden, D. Wigdor, J. Hillis, O. Hilliges, H. Benko
In ProceedingsProceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, New Orleans, LA, USA, 2019

Teaser-Picture

Context-Aware Online Adaptation of Mixed Reality Interfaces


Teaser-Picture

AudioTouch: Minimally Invasive Sensing of Micro-Gestures via Active Bio-Acoustic Sensing


Teaser-Picture

Interactive Hand Pose Estimation using a Stretch-Sensing Soft Glove


Teaser-Picture

Deformation Capture via Self-Sensing Capacitive Arrays

AuthorsO. Glauser, D. Panozzo, O. Hilliges, O. Sorkine-Hornung
In ACM Transactions on Graphics (TOG) (Volume: 38, Issue: 2), 2019

Teaser-Picture

Unpaired Pose Guided Human Image Generation


Teaser-Picture

Video-based Prediction of Hand-grasp Preshaping with Application to Prosthesis Control

AuthorsL. Taverne*, M. Cognolato*, T. Bützer, R. Gassert, O. Hilliges
In ProceedingsInternational Conference on Robotics and Automation (ICRA), 2019
* These two authors contributed equally to this work.

Teaser-Picture

Demonstration-Guided Deep Reinforcement Learning of Control Policies for Dexterous Human-Robot Interaction


Teaser-Picture

STCN:Stochastic Temporal Convolutional Networks


Teaser-Picture

TacTiles: Dual-mode Low-power Electromagnetic Actuators for Rendering Continuous Contact and Spatial Haptic Patterns in VR

AuthorsV. Vechev*, J. Zarate*, D. Lindlbauer, R. Hinchet, H. Shea, O. Hilliges
In Proceedings2019 IEEE Virtual Reality (VR), Mar 2019
* These two authors contributed equally to this work.