Information Theory in Artificial Intelligence: An Introduction

By Mohamed Medhat Gaber

This book is a comprehensive introduction to the application of information theory in artificial intelligence (AI). It starts with the fundamentals of information theory, including its historical context and key contributors, such as Claude Shannon and Norbert Wiener. The core concepts of entropy, mutual information, and Kullback-Leibler divergence are explained, along with their mathematical foundations and properties. The book also covers practical applications of these concepts in communication systems and data compression. Moving into the intersection of information theory and machine learning, the book explores how information-theoretic measures are utilised for feature selection, model evaluation, and the construction of decision trees. It delves into advanced algorithms such as the Information Bottleneck method and discusses the role of entropy and mutual information in neural networks, clustering, and variational inference. The book highlights how these principles aid in understanding and improving machine learning algorithms and their performance. The later chapters address the application of information theory in natural language processing (NLP), discussing language models, text compression, and machine translation. Advanced topics like information geometry, differential entropy, quantum information theory, and adversarial information theory are also explored. The book concludes with a discussion on future directions and emerging trends in information theory for AI, including its role in explainable AI, information privacy, and ethical considerations.

• Format: Paperback
• Language: English
• ISBN: B0DBG4ZF4L
• ISBN13: 9798334327016
• Release Date: July 2024
• Publisher: Independently published
• Length: 116 Pages
• Dimensions: 9.2" x 0.3" x 6.1"