Overview
Recent developments in neural network (aka “deep learning”) have drastically advanced the performance of machine perception systems in a variety of areas including drones, self-driving cars and intelligent UIs. This course is a deep dive into details of the deep learning algorithms and architectures for a variety of perceptual tasks.
Announcements
- 01.02.2021
- The first lecture will take place on Thursday, 25th of February. The Wednesday lecture starts in the second week of the semester.
- 22.02.2021
- Both the lecture and the exercise will be held virtually via Zoom for the entirety of the semester. The sessions will be recorded and made available for offline viewing.
- 22.02.2021
- Please sign up to Piazza.
Learning Objectives
Students will learn about fundamental aspects of modern deep learning approaches for perception. Students will learn to implement, train and debug their own neural networks and gain a detailed understanding of cutting-edge research in learning-based computer vision, robotics and HCI. The final project assignment will involve training a complex neural network architecture and applying it on a real-world dataset.
The core competency acquired through this course is a solid foundation in deep-learning algorithms to process and interpret human input into computing systems. In particular, students should be able to develop systems that deal with the problem of recognizing people in images, detecting and describing body parts, inferring their spatial configuration, performing action/gesture recognition from still images or image sequences, also considering multi-modal data, among others.
Schedule
Subject to change. Materials only available from within ETH network.
| Wk. | Date | Content | Material | Exercise Session |
|---|---|---|---|---|
| 1 | 25.02 |
Deep Learning IntroductionClass content & admin, |
slides slides (annotated) recording Perceptron Visualization Notebook |
Tutorial Implement your own MLP slidesrecording XOR Notebook XOR Solutions Eye-Gaze Notebook Eye-Gaze Solutions |
| 2 | 03.03 04.03 |
Training Neural NetworksBackpropagation |
slides part I recording part I slides part II recording part II |
Tutorial Linear Regr. slidesrecording Linear Regression Notebook Pen & Paper Backprop. exerciseexercise solution |
| 3 | 10.03. 11.03. |
Convolutional Neural Networks |
slides Additional material:
|
Tutorial CNNs in Pytorch slidesrecording CNN Notebook Pen & Paper CNN exerciseexercise solution |
| 4 | 17.03. |
Fully Convolutional Neural Networks |
slides
recording |
|
| 4 | 18.03. |
Recurrent Neural NetworksLSTM, GRU, Backpropagation through time |
Tutorial RNNs in Pytorch slidesrecording RNN Notebook Pen & Paper RNN exercisesolution |
|
| 5 | 24.03. 25.03 |
Generative Models Pt. I: Latent Variable ModelsVariational Autoencoders, etc. |
slides pt. I
Additional material:
|
Class Tips for Training I slidesrecording Pen & Paper VAE exerciseexercise solution |
| 6 | 31.03. 01.04. |
Generative Models Pt. II: Autoregressive ModelsPixelCNN, PixelRNN, WaveNet, Stochastic RNNs |
slides pt. I
slides pt. I (annotated) recording pt. I slides pt. II slides pt. II (annotated) recording pt. II |
Class Tips for Training II slidesrecording Pen & Paper AR exerciseexercise solution |
| 7 | 07.04. 08.04. |
-- No Class (Easter) -- |
||
| 8 | 14.04. 15.04. |
Generative Models Pt. III: Normalizing Flows and Invertible Neural Networks
|
slides NF Pt. I
Additional material:
|
Pen & Paper NF exerciseexercise solution |
| 9 | 21.04. 22.04. |
Generative Models Pt. III: Implicit ModelsGenerative Adversarial Networks & Co |
slides GAN Pt. I
slides GAN Pt. I (anno.) recording pt. I slides GAN Pt. II slides GAN Pt. II (anno.) recording pt. II |
Pen & Paper GAN exerciseexercise solution |
| 10 | 28.04 29.04. |
Graph Neural Networks
|
slides GNN
GNN recording pt. I GNN recording pt. II |
Pen & Paper GNN exerciseexercise solutions |
| 11 | 05.05 06.05. |
Parametric Human Body Models
|
slides HBM
HBM recording pt. I HBM recording pt. II HBM recording pt. III |
|
| 12 | 12.05 |
Human Body Model Application (Optimization-based)
|
slides HBM recording |
|
| 12 | 13.05. |
-- No Class (Ascension Day) -- |
||
| 13 | 19.05 |
Human Body Model Application (Deep Learning-based)
|
slides HBM recording |
Pen & Paper HMB exerciseexercise solution |
| 13 | 20.05 |
Hand Pose / Eye Gaze Estimation
|
slides Hand Pose Additional material:
|
|
| 14 | 26.05. 27.05. |
Reinforcement Learning |
slides RL pt. I recording pt. I-a recording pt. I-b slides RL pt. II slides RL pt. II (annotated) recording pt. II |
Pen & Paper RL exerciseexercise solutions |
| 15 | 02.06. |
Recent Research (VLG) |
recording | |
| 15 | 03.06. |
Recent Research (AIT) |
slides Research Overview recording |
Exercise Sessions
Please refer to the above schedule once available for an overview of the planned exercise slots. We will have three different types of activities in the exercise sessions:
- Tutorial: Interactive programming tutorial in Python taught by a TA. Code will be made available.
- Class: Lecture-style class taught by a TA to give you some tips on how to train your neural network in practice.
- Pen & Paper: Pen & paper exercises that are not graded but are helpful to prepare for the written exam. Solutions will be published on the website a week after the release and discussed in the exercise session if desired.
Project
Overview
There will be a multi-week project that gives you the opportunity to have some hands-on experience with training a neural network for a concrete application. The project is to be completed in groups of three and will be graded. The project grade counts 40 % towards your final grade.
The project grade will be determined by two factors: 1) a competitive part based on how well your model fairs compared to your fellow students' models and 2) the idea/novelty/innovativeness of your approach based on a written report to be handed in by the project deadline. For each project there will be baselines available that guarantee a certain grade for the competitive part if you surpass them. The competition will be hosted on a online platform - more details will be announced here.
Exam
To give you a rough idea what to expect for the exam, we release a mock exam which you can download here: