Introduction to Classification

The goal of classification is always to get a statement about the membership of an unknown substance (represented by some physical properties, i.e. a spectrum) to a particular class (defined by the user). In the best case this assignment of membership is accompanied by the probability of belonging to this class. As the raw data is normally not well-suited for establishing a classifier, we need to do some pre-processing steps before we can start the creation of a classifier. In general, the creation of a classifier is usually done along the following lines:

1. pre-processing: normalization of spectra, removal of outliers, amplitude compression, ....
2. augmenting the feature space: in many cases it can be quite helpful to derive additional features from the raw data and append these features to the input data space.
3. feature selection: in many cases the training of a classifier can be much more efficient if the dimensionality of the input space (= the number of features) is reduced to a set of significant features before training the classifier.
4. training of the classifier: the training of the classifier depends largely on the chosen classification model; some models require to adjust so called "hyper parameters" in addition to the actual training procedure.
5. evaluation of the classifier: after having created a classifier we have to evaluate its performance. Please note that the evaluation is a necessary step which must not be omitted in order to avoid overfitting.
6. application of the classifier: finally we can apply the trained classifier. Please keep in mind that the unknown data have to be subjected to the same pre-processing steps as the training data.

Pre-Processing

Augmenting the Feature Space

The introduction of additional features should be governed by spectroscopic and theoretic knowledge. So, for example, if we know the spectral pattern of a certain species we could introduce a new variable which simply represents the correlation of the sample spectrum to the spectral pattern of the particular species. Thus we introduce a new variable which is sensitive to this species and which might help to discriminate this species from other substances.

In contrast to the knowledge-based approach the augmentation of the feature space can be driven to its limits by algorithmically generating a huge set of features followed by FFX (fast function extraction) [Lit.]. Another approach is the use of deep learning approaches to generate new features [Helin 2021]

Feature selection

... to be completed ... and dimensionality reduction and selectivity enhancement

Training of the Classifier

Thus the training of a classifier can be seen as parametrization of an algorithm such, that the algorithm is able to correctly assign a spectrum to a binary class property.

In general terms, we have a p-dimensional space R^p on the input and a q-dimensional space S^q on the output or target side. The dimensionality of the input space (p) is typically between 10 and 10000, the dimensionality of the target space (q) is 1 for a two-class discrimination and nc for a multi-class problem (with nc being the number of classes to be distinguished, see below for details). In order to establish a classifier we have to use a set of observations with known and correct class assignments which fill (in a geometric sense) both the R and the S space. This set of correct observations is commonly called the "ground truth":

The ground truth then has to be split into two or three non-overlapping subsets, which serve different purposes: the training set, the test set and - sometimes - a monitoring set. The training set is - as the name implies - used to parametrize the classifier. The test set is used for the evaluation of the classifier performance and the monitoring dataset is used with some classification methods to monitor the progress of the training (as, for example, with ANNs).

Please note that quality of the ground truth dataset is of utmost importance. If the training dataset contains incorrect class assignments or problematic spectra the resulting classifier will be inferior. However, there is a simple means to check the ground truth dataset if the number of incorrect samples is low (given that the classification problem can be solved by the chosen classifier model): apply full cross validation and compare the results of the estimated classes with the classes recorded in the ground truth dataset. Those observations whose estimated class value differ the most are likely to be candidates for re-inspection of the ground truth.

Now the art of creating a classifier lies in the parametrization of the selected classification model. --_> parameters and hyper parameters

What to do in a multi-class case?

--> indicator variables --> OVA and similar decision rules

... see hyperspectral -> Coimbra ... to be completed

What are hyperparameters: hyperparameters are parameters of the classification model which are not found by the training procedure but which nevertheless affect the model performance. Hyperparameters can be divided into two groups: on the one hand hyperparameters might influence the quality and speed of the learning process but have no effect on the resulting classifier, and on the other hand hyperparameters can influence the trained classifier as well. An example for hyperparameters is the number of hidden neurons in ANNs, or the learning rate of the training algorithm. The first greatly influences the resulting classifier while the latter does not or to almost no extent.

In general hyperparameters make the creation of a classifier considerably more time consuming because the optimal hyperparameters can only be determined by scanning the hyperparameter space, repeating the training for each set of hyperparameters.

Evaluation

Metric	Definition
TP rate, recall	TP/(TP+FN)
FP rate, fall-out	FP/(FP+TN)
Accuracy	(TP+TN)/(TP+TN+FP+FN)
MCC (Matthews Correlation Coefficient	(TP*TN-FP*FN)/sqrt((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))
AUC	area under the ROC curve