Decomposition of a Single ObservationΒΆ
The entropy of an observation of a single symbol from a time series can be
decomposed into several parts. After the observation of many symbols, we may
come to understand the structure therein, and so our next symbol does not stand
apart, but rather shares some information with our past observations. This
shared amount is known as the total correlation rate, , because
it is the asymptotic rate of the block total correlation curve. Carving
out of our single symbol entropy leaves us with the
“surprising” amount of information in the observation, the entropy rate
, known as such because it is the rate at which the block entropy
curve grows asymptotically.
Abdallah and Plumbley [abdallah2010] defined the predictive information rate as
a measure of complexity in discrete time series. Their particular use case is
finding a measure of “interestingness” for music, but the measure does stand on
its own outside of music. In fact we can decompose into two pieces:
a part which stands on its own and doesn’t influence nor is influenced by past
or future observations, and a part which is shared with future structure. The
parts are known as the residual entropy rate,
, and the predictive
information rate,
, respectively.
