Effective Rank and Continuous Spectrum
Dash
Tutorials
Issues
Blog
Sign In

Effective Rank and Continuous Spectrum

written by Ge Yang

Effective Rank

Roy et al1 provides the following definition for the effective rank:

Consider a matrix AA with a singular value decomposition (SVD)

A=UDVA = U D V

where D is a M×NM \times N diagonal matrix with the singular values

σ1≥σ2≥⋯≥σQ≥0\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_Q \geq 0

where Q=min⁡{M,N}Q = \min\{M, N\}.

Definition 1 (Nuclear norm) The nuclear norm of the matrix AA is defined as

∥σ∥1=∑i=1Qσi.\Vert \sigma \Vert_1=\sum_{i=1}^Q \sigma_i.

Then the sequence of singular values gives rise to a nominal distribution

pi=σi∥σ∥1 for i∈{1,2,…,Q}p_i = \frac{\sigma_i}{\Vert \sigma \Vert_1} \text{ for } i \in \{1, 2, \dots, Q\}

Definition 2 (Effective rank) The effective rank of a matrix AA is defined as the volume

erank(A)=exp⁡H(σi∥σ∥1).\mathrm{erank}(A) = \exp{H\left(\frac{\sigma_i}{\Vert \sigma \Vert_1}\right)}.

where HH is the Shannon entropy of the nominal distribution given by the matrix spectrum.

Entropy of A Continuous Spectrum

The effective rank given by Roy et al is only defined on discrete spectrum. We know that computing the entropy of a continuous distribution require special care2.

Spectral Entropy in Finite Sampling Window

An additional consideration when computing the spectral entropy on a discrete spectrum, is that the windowing function produced by applying discrete Fourier transform in a finite message window imprints a spectral floor that contributes to the spectral distribution. We can subtract the spectral entropy of this windowing function from the spectral entropy of the discrete Fourier spectrum, s.t. the entropy of the discrete spectrum is 00 when there is just a single component.

H(S)=H(S)−H(w)H(S) = H(S) - H(w)

Acknowledgements:

Leave the Acknowledgements here.

References

  1. Roy, O. and Vetterli, M. (2007) ‘The effective rank: A measure of effective dimensionality’, in 2007 15th European Signal Processing Conference, pp. 606–610. Available at: http://ieeexplore.ieee.org/document/7098875/↩
  2. Ge Yang, 'Differential Entropy and Invariant Entropy Measure', Available at: https://www.episodeyang.com/blog/2022/10-23/entropy_continuous/↩