# Course Details

Course Information Package

Course Unit TitleINFORMATION THEORY
Course Unit CodeAEEE515
Course Unit DetailsMSc Electrical Engineering (Technical Electives) -
Number of ECTS credits allocated7
Learning Outcomes of the course unitBy the end of the course, the students should be able to:
1. Calculate relative entropies and mutual information. State the chain rules for entropy, relative entropy, and mutual Information. Appraise Jensen’s inequality and its consequences. Explain Fano’s inequality and the asymptotic equipartition property theorem.
2. Evaluate various codes and interpret the nature of optimal codes. Estimate bounds on the optimal code length. Explain Kraft’s inequality for uniquely decodable codes. Examine Huffman codes. Define optimality of Huffman codes and Shannon–Fano–Elias coding.
3. Calculate channel capacity. Describe noiseless binary channels and noisy channels with non-overlapping outputs. Examine the various types of channels such as the binary symmetric channel, the binary erasure channel and the symmetric channels. State the channel coding theorem. Examine zero-error codes and Hamming codes.
4. Formulate the coding theorem for Gaussian channels and band-limited channels and parallel Gaussian channels.
5. Describe multiple-user channels and Gaussian multiple-access channel with m users. Examine the Gaussian broadcast channel, the Gaussian relay channel and the Gaussian interference channel. Assess the uncertainty on Shannon’s reliable data transmission blocks.
Mode of DeliveryFace-to-face
PrerequisitesNONECo-requisitesNONE
Recommended optional program componentsNONE
Course Contents

Entropy, Relative Entropy, and Mutual Information

Entropy. Joint entropy and conditional entropy. Relative entropy and mutual information. Relationship between entropy and mutual information. Chain rules for entropy, relative entropy, and mutual Information. Jensen’s inequality and its consequences. Log sum inequality and its applications. Sufficient statistics. Fano’s inequality. Asymptotic equipartition property theorem.

Data Compression

Examples of codes. Kraft inequality. Optimal codes. Bounds on the optimal code length. Kraft inequality for uniquely decodable codes. Huffman codes. Optimality of Huffman codes. Shannon–Fano–Elias coding.

Channel Capacity

Examples of channel capacity. Noiseless binary channel. Noisy channel with non-overlapping outputs. Binary symmetric channel. Binary erasure channel. Symmetric channels. Properties of channel capacity. Channel coding theorem. Zero-error codes. Hamming codes.

Gaussian Channel

Gaussian channel. Converse to the coding theorem for Gaussian channels. Band-limited channels. Parallel Gaussian channels.

Network Information Theory

Gaussian multiple-user channels. Gaussian multiple-access channel with m users. Gaussian broadcast channel. Gaussian relay channel. Gaussian interference channel. Uncertainty on Shannon’s reliable data transmission blocks.