The Curse of Dimensionality

In a typical case (for example, looking at a Raman spectrum with a resolution of 3 wave numbers we get approx. 1100 intensity values) which leads to a 10¹¹⁰⁰-dimensional space. This huge space is populated by 10⁴ observations (pixels) which is near to nothing in comparison to the available space. Of course this crude estimation has to be refined a little bit, since we did not take into account the correlation of the variables within in individual bands of a spectrum. If we assume the width of typical Raman bands being about 40 wave numbers we still end up with about 90 variables leading to a 10⁹⁰ dimensional space - which is still almost empty due to its hugeness.

So what can we do against this curse of dimensionality? We have basically two options: (1) reduce the dimensionality of the space, or (2) increase the number of observations (pixels). As the second option is not feasible for practical reasons (we cannot measure that much of spectra in order to get a non-empty data space, nor we can process that much of data - remember the age of the universe is estimated as roughly 4*10¹⁷ seconds). So the only way to improve this unfavorable situation is to reduce the dimensionality.

The most common approach to reduce the dimensionality of a data space is to perform variable selection. However, the information we are be interested in, might not be represented by individual variables but by a complex combination of several variables. Thus in many cases variable selection will not deliver the optimum solution (in fact there are situations - especially in mass spectrometry - where variable selection does not provide a solution at all).

The solution to this difficult combination of the curse of dimensionality and the impossibility to find an optimum set of variables for a particular problem is the introduction of chemical/physical knowledge. By introducing chemical knowledge we implicitly transform the data space and reduce its dimensionality. How can this be achieved? Read on here....