MSc in Electrical Engineering

Course Information Package

Course Unit TitleINFORMATION THEORY
Course Unit CodeAEEE515
Course Unit DetailsMSc Electrical Engineering (Technical Electives) -
Number of ECTS credits allocated7
Learning Outcomes of the course unitBy the end of the course, the students should be able to:
  1. Calculate relative entropies and mutual information. State the chain rules for entropy, relative entropy, and mutual Information. Appraise Jensen’s inequality and its consequences. Explain Fano’s inequality and the asymptotic equipartition property theorem.
  2. Evaluate various codes and interpret the nature of optimal codes. Estimate bounds on the optimal code length. Explain Kraft’s inequality for uniquely decodable codes. Examine Huffman codes. Define optimality of Huffman codes and Shannon–Fano–Elias coding.
  3. Calculate channel capacity. Describe noiseless binary channels and noisy channels with non-overlapping outputs. Examine the various types of channels such as the binary symmetric channel, the binary erasure channel and the symmetric channels. State the channel coding theorem. Examine zero-error codes and Hamming codes.
  4. Formulate the coding theorem for Gaussian channels and band-limited channels and parallel Gaussian channels.
  5. Describe multiple-user channels and Gaussian multiple-access channel with m users. Examine the Gaussian broadcast channel, the Gaussian relay channel and the Gaussian interference channel. Assess the uncertainty on Shannon’s reliable data transmission blocks.
Mode of DeliveryFace-to-face
PrerequisitesNONECo-requisitesNONE
Recommended optional program componentsNONE
Course Contents

Entropy, Relative Entropy, and Mutual Information

Entropy. Joint entropy and conditional entropy. Relative entropy and mutual information. Relationship between entropy and mutual information. Chain rules for entropy, relative entropy, and mutual Information. Jensen’s inequality and its consequences. Log sum inequality and its applications. Sufficient statistics. Fano’s inequality. Asymptotic equipartition property theorem.

Data Compression

Examples of codes. Kraft inequality. Optimal codes. Bounds on the optimal code length. Kraft inequality for uniquely decodable codes. Huffman codes. Optimality of Huffman codes. Shannon–Fano–Elias coding.

Channel Capacity

Examples of channel capacity. Noiseless binary channel. Noisy channel with non-overlapping outputs. Binary symmetric channel. Binary erasure channel. Symmetric channels. Properties of channel capacity. Channel coding theorem. Zero-error codes. Hamming codes.

Gaussian Channel

Gaussian channel. Converse to the coding theorem for Gaussian channels. Band-limited channels. Parallel Gaussian channels.

Network Information Theory

Gaussian multiple-user channels. Gaussian multiple-access channel with m users. Gaussian broadcast channel. Gaussian relay channel. Gaussian interference channel. Uncertainty on Shannon’s reliable data transmission blocks.
Recommended and/or required reading:
Textbooks
  • T. M. Cover and J. A. Thomas, Elements of Information Theory, Second Edition, Wiley, 2006.
References
  • R. W. Yeung, Information Theory and Network Coding (Information Technology: Transmission, Processing and Storage), Springer Science, 2008.
  • R. W. Yeung, A first course in information theory, Springer Science, 2002.
  • R. E. Blahut, Principles and Practice of Information Theory, Addison-Wesley, 1987.
  • C. Shannon and W. Weaver, Mathematical Theory of Communication, University of Illinois, 1998.
Planned learning activities and teaching methodsThe teaching of the course is lecture-based (3 hours per week) in a classroom. A combination of traditional teaching with written notes on a white board and slide presentations using a projector for the presentation of the more complicated diagrams and graphs is utilized. Students are assessed continuously and their knowledge is checked through tests, assignments and the final exam. Lectures include the solution and discussion of example problems regarding the material presented. Relevant homework and assignments are given to the students for further study at their own. Due to the level and type of the course students are urged to participate in discussing the various topics and provide their opinion during problem-solving sessions. Lecture notes are compiled by students which serve to cover the main issues under consideration and serve as a guide for further reading. Students are also required to seriously use the textbook assigned to the course, in addition to other sources found either in the library or elsewhere in order to broaden their perspective on the various subjects presented in class and in the textbook.
Assessment methods and criteria
Assignments20%
Tests30%
Final Exam50%
Language of instructionEnglish
Work placement(s)NO