Information Theory

Information theory is the quantitative study of information. Information is generally knowledge of the outcome of an event, and thus probability distributions take a central role in information theory.

• Defining a Probability Distribution
  • Standard Distributions
  • Log Distributions
  • Symbolic Distributions
  • Generalized Distributions
• Working With Distributions
  • Normalizing & Validating
  • Converting Between Distribution Types
  • Marginalizing
  • Conditioning
• Measuring Information
  • Entropy
  • Perplexity
  • Conditional Entropy
  • Cross Entropy
  • Relative Entropy
  • Mutual Information
  • Conditional Mutual Information
  • Multivariate Mutual Information
  • Interaction Information
  • Total Correlation
  • Binding Information
  • In Summary