How to calculate the entropy rate hmu of a processΒΆ

The entropy rate hmu is defined as the entropy per symbol of infinitely long blocks of symbols. h_\mu=\lim_{L\to\infty}H(X^L)/L. Practically, it can be calculated in two ways.

First, as the asymptotic increase in block entropy per symbol when considering ever larger blocks of symbols:

h_\mu=\lim_{L\to\infty}h_\mu(L) where h_\mu(L)=H(X^L+1) - H(X^L)

The quantity h_mu(L) can be calculated in cmpy via the cmech_quantities function for a whole range of values of L:


This returns a masked array of length Lmax+1 of h_\mu(L) for L=0..Lmax. The true value of hmu to which hmu(L) converges for L to infinite is returned in multiples of L via:


This again returns a masked array of length Lmax+1 with each entry being hmu*L for L=0..Lmax. The entropy rate can then be found as:


Naturally, if only hmu is desired, it is enough to set Lmax=1.

We can also demonstrate the convergence of the block entropy H(X_0^L) against the entropy rate. With:


we obtain a masked array of the block entropy per symbol. Note the use of the masked array as it allows us to divide by block lengths of zero.

The following figure was created with the help of these commands and shows the convergence of both h_\mu(L) and H(X_0^L)/L against hmu.