Machine Perception - SS 21

Recent developments in neural network (aka “deep learning”) have drastically advanced the performance of machine perception systems in a variety of areas including drones, self-driving cars and intelligent UIs. This course is a deep dive into details of the deep learning algorithms and architectures for a variety of perceptual tasks.


Overview

eDoz Course Nr.
263-3710-00L
Lecturer
O. Hilliges, S. Tang
Assistants
A. Spurr, S. Christen, Z. Fan, M. Kaufmann, M. Mihajlovic S. Wang Y. Zhang
Lecture
Wed 13 - 14, live streamed via Zoom. Zoom Link Lecture
Thu 12 - 14, live streamed via Zoom.
Exercises
Thu 14 - 16, live streamed via Zoom. Zoom Link Exercise
Fri 14 - 16, live streamed via Zoom.
Credits
8 ECTS
Recordings
The lecture recordings will be made available.
Contact
Contact
Please address all questions regarding content, organisation etc. on Piazza. Please sign up to the forum using this link. The forum is closely monitored by us. Due to organisational reasons, we will not be able to respond to direct e-mails.



Announcements

01.02.2021
The first lecture will take place on Thursday, 25th of February. The Wednesday lecture starts in the second week of the semester.
22.02.2021
Both the lecture and the exercise will be held virtually via Zoom for the entirety of the semester. The sessions will be recorded and made available for offline viewing.
22.02.2021
Please sign up to Piazza.



Learning Objectives

Students will learn about fundamental aspects of modern deep learning approaches for perception. Students will learn to implement, train and debug their own neural networks and gain a detailed understanding of cutting-edge research in learning-based computer vision, robotics and HCI. The final project assignment will involve training a complex neural network architecture and applying it on a real-world dataset.

The core competency acquired through this course is a solid foundation in deep-learning algorithms to process and interpret human input into computing systems. In particular, students should be able to develop systems that deal with the problem of recognizing people in images, detecting and describing body parts, inferring their spatial configuration, performing action/gesture recognition from still images or image sequences, also considering multi-modal data, among others.




Schedule

Subject to change. Materials only available from within ETH network.

Wk.Date ContentMaterial Exercise Session
1 25.02
Deep Learning Introduction

Class content & admin,
Feedforward Networks,
Representation Learning

slides
slides (annotated)
recording

Perceptron Visualization Notebook

Tutorial Implement your own MLP

slides
recording
XOR Notebook
XOR Solutions
Eye-Gaze Notebook
Eye-Gaze Solutions
2 03.03
04.03
Training Neural Networks

Backpropagation

slides part I
recording part I

slides part II
recording part II

Tutorial Linear Regr.

slides
recording
Linear Regression Notebook

Pen & Paper Backprop.

exercise
exercise solution
3 10.03.
11.03.
Convolutional Neural Networks

slides
recording part I
recording part II

Additional material:
RSVP
Cortical Neuron

Tutorial CNNs in Pytorch

slides
recording
CNN Notebook

Pen & Paper CNN

exercise
exercise solution
4 17.03.
Fully Convolutional Neural Networks
slides
recording

4 18.03.
Recurrent Neural Networks

LSTM, GRU, Backpropagation through time

slides
recording

Tutorial RNNs in Pytorch

slides
recording
RNN Notebook

Pen & Paper RNN

exercise
solution
5 24.03.
25.03
Generative Models Pt. I: Latent Variable Models

Variational Autoencoders, etc.

slides pt. I
slides pt. I (annotated)
recording pt. I

slides pt. II
slides pt. II (annotated)
recording pt. II

Additional material:
Denoising AE
Image Super Res.
Celeste
Ghosh et. al (2017)
Ghosh et. al (2017)
Chemical Design
World Models

Class Tips for Training I

slides
recording

Pen & Paper VAE

exercise
exercise solution
6 31.03.
01.04.
Generative Models Pt. II: Autoregressive Models

PixelCNN, PixelRNN, WaveNet, Stochastic RNNs

slides pt. I
slides pt. I (annotated)
recording pt. I

slides pt. II
slides pt. II (annotated)
recording pt. II

Class Tips for Training II

slides
recording

Pen & Paper AR

exercise
exercise solution
7 07.04.
08.04.
-- No Class (Easter) --
8 14.04.
15.04.
Generative Models Pt. III: Normalizing Flows and Invertible Neural Networks

slides NF Pt. I
slides NF Pt. I (anno.)
recording pt. I

slides NF Pt. II
slides NF Pt. II (anno.)
recording pt. II

Additional material:
RealNVP
NICE
GLOW
NeuralODE
FFJORD
SRFlow
StyleFlow
StyleFlow (video)
C-Flow
C-Flow (video)

Pen & Paper NF

exercise
exercise solution
9 21.04.
22.04.
Generative Models Pt. III: Implicit Models

Generative Adversarial Networks & Co

slides GAN Pt. I
slides GAN Pt. I (anno.)
recording pt. I

slides GAN Pt. II
slides GAN Pt. II (anno.)
recording pt. II

Pen & Paper GAN

exercise
exercise solution
10 28.04
29.04.
Graph Neural Networks

slides GNN
GNN recording pt. I

GNN recording pt. II

Pen & Paper GNN

exercise
exercise solutions
11 05.05
06.05.
Parametric Human Body Models

slides HBM
HBM recording pt. I
HBM recording pt. II
HBM recording pt. III

12 12.05
Human Body Model Application (Optimization-based)

slides HBM
recording
12 13.05.
-- No Class (Ascension Day) --
13 19.05
Human Body Model Application (Deep Learning-based)

slides HBM
recording

Pen & Paper HMB

exercise
exercise solution
13 20.05
Hand Pose / Eye Gaze Estimation

slides Hand Pose
slides Eye Gaze
recording

Additional material:
SMPL
Embodied Hands
Boukhayma et. al (2019)
Kulon et. al (2020)
Iqbal et. al (2018)

14 26.05.
27.05.
Reinforcement Learning
slides RL pt. I
recording pt. I-a
recording pt. I-b
slides RL pt. II
slides RL pt. II (annotated)
recording pt. II

Pen & Paper RL

exercise
exercise solutions
15 02.06.
Recent Research (VLG)

recording
15 03.06.
Recent Research (AIT)

slides Research Overview
recording



Exercise Sessions

Please refer to the above schedule once available for an overview of the planned exercise slots. We will have three different types of activities in the exercise sessions:

  1. Tutorial: Interactive programming tutorial in Python taught by a TA. Code will be made available.
  2. Class: Lecture-style class taught by a TA to give you some tips on how to train your neural network in practice.
  3. Pen & Paper: Pen & paper exercises that are not graded but are helpful to prepare for the written exam. Solutions will be published on the website a week after the release and discussed in the exercise session if desired.




Project

Overview

There will be a multi-week project that gives you the opportunity to have some hands-on experience with training a neural network for a concrete application. The project is to be completed in groups of three and will be graded. The project grade counts 40 % towards your final grade.

The project grade will be determined by two factors: 1) a competitive part based on how well your model fairs compared to your fellow students' models and 2) the idea/novelty/innovativeness of your approach based on a written report to be handed in by the project deadline. For each project there will be baselines available that guarantee a certain grade for the competitive part if you surpass them. The competition will be hosted on a online platform - more details will be announced here.

Exam

To give you a rough idea what to expect for the exam, we release a mock exam which you can download here: