Point-based metrics

Warning

It is known that the point-adjusted metrics heavily overestimate the performance of anomaly detectors. It is therefore not recommended to solely rely on those metrics to evaluate a model. These metrics were implemented for reproducibility of existing works.

class dtaianomaly.evaluation.Precision

Computes the Precision score.

Precision measures how accurately the model identifies anomalies. It reflects the proportion of detected anomalies that are truly abnormal. This is particularly important when the cost of false positives (normal events incorrectly flagged as anomalies) is high.

Mathematically, precision is the ratio of true positives (correctly identified anomalies) to all predicted positives, which includes both true anomalies and false positives (normal events mistakenly flagged as anomalies). It can be expressed as:

\[\text{Precision} = \frac{\text{True Anomalies}}{\text{True Anomalies} + \text{False Positives}}\]

A high precision in anomaly detection indicates that the model generates few false alarms, ensuring that most flagged anomalies are truly abnormal events. However, it does not measure how many anomalies were actually identified.

class dtaianomaly.evaluation.Recall

Computes the Recall score.

Recall measures the model’s ability to correctly identify all actual anomalies. It tells us the proportion of true anomalies that were successfully detected by the model. In an anomaly detection system, recall answers the question: “Of all the anomalies that occurred, how many did the model detect?” A high recall is especially important when missing actual anomalies (false negatives) could have severe consequences.

Mathematically, recall is the ratio of true positives (correctly identified anomalies) to all actual positives, which includes both true anomalies and false negatives (missed anomalies). It can be expressed as:

\[\text{Recall} = \frac{\text{True Anomalies}}{\text{True Anomalies} + \text{False Negatives}}\]

A high recall ensures that most anomalies are detected, but it doesn’t account for how many false positives (normal events incorrectly flagged as anomalies) were generated, which is handled by precision.

class dtaianomaly.evaluation.FBeta(beta: (<class 'float'>, <class 'int'>) = 1)

Computes the \(F_\beta\) score.

The \(F_\beta\) combines both precision and recall into a single value. It provides a balanced evaluation of a model’s performance, especially in anomaly detection, where there is often a trade-off between catching all anomalies (high recall) and minimizing false alarms (high precision). The parameter \(\beta\) controls the balance between precision and recall. A \(\beta > 1\) gives more weight to recall, useful when missing anomalies is costly, while \(\beta < 1\) emphasizes precision, reducing false positives.

The \(F_\beta\) score is the harmonic mean of precision and recall. It can be expressed as:

\[F_\beta = \frac{(1 + \beta^2) \text{tp}} {(1 + \beta^2) \text{tp} + \text{fp} + \beta^2 \text{fn}}\]

A high \(F_\beta\) score indicates a good balance between detecting actual anomalies and minimizing false positives.

Parameters:

beta (int, float, default=1) – Desired beta parameter.

See also

Precision

Compute the Precision score.

Recall

Compute the Recall score.

class dtaianomaly.evaluation.PointAdjustedPrecision

Compute the point-adjusted precision: first point-adjust the predicted anomaly scores, after which the precision is computed.

For given binary anomaly predictions and ground truth anomaly labels, point-adjusting will treat any sequence of consecutive ground truth anomalies as anomalous events. If any of the observations in such an event has been detected, then we say that the anomaly has been detected. In this case, all predictions in the anomalous event are set to 1, thereby indicating that the method predicted an anomaly.

metric

The Precision-object used for computing the precision, after the prediction has been point adjusted. Note that the object should not be passed to the constructor.

Type:

Precision

See also

Precision

Compute the standard, not point-adjusted precision.

class dtaianomaly.evaluation.PointAdjustedRecall

Compute the point-adjusted recall: first point-adjust the predicted anomaly scores, after which the recall is computed.

For given binary anomaly predictions and ground truth anomaly labels, point-adjusting will treat any sequence of consecutive ground truth anomalies as anomalous events. If any of the observations in such an event has been detected, then we say that the anomaly has been detected. In this case, all predictions in the anomalous event are set to 1, thereby indicating that the method predicted an anomaly.

metric

The Recall-object used for computing the precision, after the recall has been point adjusted. Note that the object should not be passed to the constructor.

Type:

Recall

See also

Recall

Compute the standard, not point-adjusted recall.

class dtaianomaly.evaluation.PointAdjustedFBeta(beta: (<class 'float'>, <class 'int'>) = 1)

Compute the point-adjusted \(F_\beta\): first point-adjust the predicted anomaly scores, after which the \(F_\beta\) is computed.

For given binary anomaly predictions and ground truth anomaly labels, point-adjusting will treat any sequence of consecutive ground truth anomalies as anomalous events. If any of the observations in such an event has been detected, then we say that the anomaly has been detected. In this case, all predictions in the anomalous event are set to 1, thereby indicating that the method predicted an anomaly.

Parameters:

beta (int, float, default=1) – Desired beta parameter.

metric

The FBeta-object used for computing the precision, after the \(F_\beta\): has been point adjusted. Note that the object should not be passed to the constructor.

Type:

FBeta

See also

FBeta

Compute the standard, not point-adjusted \(F_\beta\).