Evaluation module

This module contains functionality to evaluate performance of an anomaly detector. It can be imported as follows:

>>> from dtaianomaly import evaluation

Custom evaluation metrics can be implemented by extending Metric or ProbaMetric. The former expects predicted “decisions” (anomaly or not), the latter predicted “scores” (more or less anomalous). This distinction is important for later use in a Worfklow.

class dtaianomaly.evaluation.Metric[source]

compute(y_true: ndarray, y_pred: ndarray) → float[source]

Computes the performance score.

Parameters:

y_true (array-like of shape (n_samples)) – Ground-truth labels.
y_pred (array-like of shape (n_samples)) – Predicted anomaly scores.

Returns:

score – The alignment score of the given ground truth and prediction, according to this score.

Return type:

float

Raises:

ValueError – When inputs are not numeric “array-like”s
ValueError – If shapes of y_true and y_pred are not of identical shape

class dtaianomaly.evaluation.BinaryMetric[source]: A metric that takes as input binary anomaly labels.

class dtaianomaly.evaluation.ProbaMetric[source]: A metric that takes as input continuous anomaly scores.

class dtaianomaly.evaluation.ThresholdMetric(thresholder: Thresholding, metric: BinaryMetric)[source]

Wrapper to combine a BinaryMetric object with some thresholding, to make sure that it can take continuous anomaly scores as an input. This is done by first applying some thresholding to the predicted anomaly scores, after which a binary metric can be computed.

Parameters:

thresholder (Thresholding) – Instance of the desired Thresholding class
metric (Metric) – Instance of the desired Metric class

class dtaianomaly.evaluation.Precision[source]

Computes the Precision score.

Precision measures how accurately the model identifies anomalies. It reflects the proportion of detected anomalies that are truly abnormal. This is particularly important when the cost of false positives (normal events incorrectly flagged as anomalies) is high.

Mathematically, precision is the ratio of true positives (correctly identified anomalies) to all predicted positives, which includes both true anomalies and false positives (normal events mistakenly flagged as anomalies). It can be expressed as:

\[\text{Precision} = \frac{\text{True Anomalies}}{\text{True Anomalies} + \text{False Positives}}\]

A high precision in anomaly detection indicates that the model generates few false alarms, ensuring that most flagged anomalies are truly abnormal events. However, it does not measure how many anomalies were actually identified.

class dtaianomaly.evaluation.Recall[source]

Computes the Recall score.

Recall measures the model’s ability to correctly identify all actual anomalies. It tells us the proportion of true anomalies that were successfully detected by the model. In an anomaly detection system, recall answers the question: “Of all the anomalies that occurred, how many did the model detect?” A high recall is especially important when missing actual anomalies (false negatives) could have severe consequences.

Mathematically, recall is the ratio of true positives (correctly identified anomalies) to all actual positives, which includes both true anomalies and false negatives (missed anomalies). It can be expressed as:

\[\text{Recall} = \frac{\text{True Anomalies}}{\text{True Anomalies} + \text{False Negatives}}\]

A high recall ensures that most anomalies are detected, but it doesn’t account for how many false positives (normal events incorrectly flagged as anomalies) were generated, which is handled by precision.

class dtaianomaly.evaluation.FBeta(beta: (<class 'float'>, <class 'int'>) = 1)[source]

Computes the \(F_\beta\) score.

The \(F_\beta\) combines both precision and recall into a single value. It provides a balanced evaluation of a model’s performance, especially in anomaly detection, where there is often a trade-off between catching all anomalies (high recall) and minimizing false alarms (high precision). The parameter \(\beta\) controls the balance between precision and recall. A \(\beta > 1\) gives more weight to recall, useful when missing anomalies is costly, while \(\beta < 1\) emphasizes precision, reducing false positives.

The \(F_\beta\) score is the harmonic mean of precision and recall. It can be expressed as:

\[F_\beta = \frac{(1 + \beta^2) \text{tp}} {(1 + \beta^2) \text{tp} + \text{fp} + \beta^2 \text{fn}}\]

A high \(F_\beta\) score indicates a good balance between detecting actual anomalies and minimizing false positives.

Parameters:: beta (int, float, default=1) – Desired beta parameter.