Information Theory (V2/Ü1)

The information on this website is also available in the Lehr- und Lernportal der RWTH Aachen. Every participant should sign up there.

Links to Campus Office

Lecture
Exercise

Contents

• Basics: entropy, Kulbach-Leibler divergence, mutual information, capacity for absolutely-continuous distributions
• Channels and their capacity: complex Gaussian, parallel, MIMO, with feedback and memory, quantization and censoring
• Rate distortion theory: achievability, computational aspects
• Network information theory: multiple user channels, relay channels, broadcast channels, source coding with side information
• Information theory for neural networks (NN) and learning: single neuron classifier, information flow in NN, Boltzmann machines

Prerequisites

Basic knowledge of probability theory and stochastic processes as given, e.g., in Theoretische Informationstechnik I. Basic knowledge of linear and convex optimization. Fundamentals of matrix analysis.

Literature

• Lecture notes for Information Theory
• T.M. Cover and J.A. Thomas. Elements of Information Theory. 2nd edition, Wiley, New York, 2006
• D.J.C. MacKay. Information Theory, Inference and Learning Algorithms, Cambridge University Press, Cambridge, 2017
• A. El Gamal, Y.H. Kim. Network Information Theory, Cambridge University Press, Cambridge, 2011

Lecture Notes

• Lecture 1 (2018-10-15)
• Lecture 2 (2018-10-22)
• Lecture 3 (2018-10-29)
• Lecture 4 (2018-11-05)
• Lecture 5 (2018-11-12)
• Lecture 6 (2018-11-19)
• Lecture 7 (2019-01-07)
• Lecture 8 (2019-01-08) - Slides
• Lecture 9 (2019-01-14)
• Lecture 10 (2019-01-15) - Slides
• Lecture 11 (2019-01-21) - Slides
• Lecture 12 (2019-01-22)
• Lecture 13 (2019-01-28) - Slides

Lecture Videos

• Lecture 1: 1080p 720p

• Lecture 2: 1080p 720p

• Lecture 3: 1080p 720p

• Lecture 4: 1080p 720p

• Lecture 5: 1080p 720p

• Lecture 6: 1080p 720p

• Lecture 7: 1080p 720p

• Lecture 8: 1080p 720p

• Lecture 9: 1080p 720p

• Lecture 10: 1080p 720p

• Lecture 11: 1080p 720p

• Lecture 12: 1080p 720p

• Lecture 13: 1080p 720p

Exercises

• Exercise 1 - Solution
• Exercise 2 - Solution
• Exercise 3 - Solution
• Exercise 4 - Solution
• Exercise 5 - Solution
• Exercise 6 - Solution
• Exercise 7 - Solution
• Exercise 8 - Solution

Consultation hour

Prof. Dr. Rudolf. Mathar upon agreement
Dr.-Ing. Gholamreza Alirezaei upon agreement
M.Sc. Emilio Balda upon agreement
M.Sc. Vimal Radhakrishnan upon agreement