Syllabus

Course Meeting Times

Lectures: 2 sessions / week, 1.5 hours / session

Course Information

This is a graduate-level introduction to mathematics of information theory. We will cover both classical and modern topics (such as finite blocklength IT and applications to statistical decision theory). Those taking Information Theory for the first time may benefit from reading the standard textbook by T. Cover and J. Thomas (see below).

Course Prerequisites

This course requires knowledge of theorem-proof exposition and probability theory, as taught in 6.042J Mathematics for Computer Science or 6.436J Fundamentals of Probability.

Course Topics

This course will cover the following topics:

1. Information measures (entropy, divergence, mutual information): Convexity, monotonicity and continuity properties. Extremization, saddle point, capacity as information radius. Variational characterizations: Donsker-Varadhan and Gelfand-Yaglom- Perez. Entropy rates and theorem of Szego.
2. Lossless data compression: Variable length and fixed length (almost lossless). Linear compression. Slepian-Wolf problem. Ergodic sources: Shannon-McMillan and Birkhoff-Khintchine theorems. Basics of universal data compression. Optimality of Lempel-Ziv.
3. Binary hypothesis testing: Bounds for finite number of samples. Asymptotics: Stein and Chernoff exponents. Large deviations: Sanov, I-projection, tilting.
4. Channel coding: Achievability and converse bounds. Asymptotics: Capacity, strong converse, error-exponents, channel dispersion. Gaussian channels (parallel, with intersymbol interference, minimal energy-per-bit, continuous time). Coding with feedback: Zero and non-zero error capacities, Schalkwijk-Kailath and variable length codes.
5. Lossy data compression: Scalar quantization and Panter-Dite approximation. Vector quantization and rate-distortion theorem. Separation principle.
6. Topics in multi-user IT: Capacity of a multiple-access channel. Gelfand-Pinsker problem. Interference channels.

Course Text

In addition to the lecture notes, students may find the following text to be of use. Some homework problems will be assigned from the text.

T. Cover, and J. Thomas. Elements of Information Theory, Second Edition. Wiley-Interscience, 2006. ISBN: 9780471241959.

Assignments and Grading

There will be weekly assigned problem sets, one take-home midterm, and one take-home final. Collaboration is allowed on the problem sets but not on the take-home exams. The final grade will be weighted as follows: