Range-based metrics

Implementations of the range-based metrics proposed by [24]. In contrast to traditional metrics such as precision and recall, these metrics take the temporal nature of time series into account. Another advantage of these metrics is that they can be tuned to better evaluate the performance of an anomaly detector for a specific domain. Specifically, you can tune the following criteria:

Existence: The importance of catching an true anomaly, even if it is only using a single predicted point, might by itself be useful for certain applications.
Position: In some cases, not only the size, but also the relative position of the predicted anomaly in the true anomaly may be important (e.g., early detection).
Cardinality: Detecting an true anomaly with only one predicted anomalous interval may be more valuable than doing so with multiple different ranges in a fragmented manner.

Tatbul et al. [2018] also proposed to allow for flexible inclusion of the size of the detected anomaly, by replacing the \(\omega()\)-function. This option is not included in our implementation.

Warning

Note that, while tuning a metric to some domain is beneficial in practical applications, this flexibility makes it difficult for a large-scale, general-purpose evaluation of multiple anomaly detectors, as you can optimize the metric for a specific application.

class dtaianomaly.evaluation.RangeBasedPrecision(delta: str | Callable[[int, int], float] = 'flat', gamma: str | Callable[[int], float] = 'reciprocal')[source]

Computes the range-based precision score [24].

The range-based precision computes a precision-score for each predicted anomalous range and then takes the average over all ranges. This precision-score consists of two parts: (1) the amount of overlap between the predicted range and the ground truth ranges, and (2) whether the predicted range overlaps with only one or multiple ground truth ranges. These components can be computed independently, and are multiplied to get a final precision-score for the range.

Parameters:

delta (str or callable, default='flat') –
Bias for the position of the predicted anomaly in the ground truth anomalous range. Valid options are:
- 'flat': Equal bias towards all positions in the ground truth anomalous range.
- 'front': Predictions that are near the front of the ground truth anomaly (i.e. early detection) have a higher weight.
- 'back': Predictions that are near the end of the ground truth anomaly (i.e. late detection) have a higher weight.
- 'middle': Predictions that are near the center of the ground truth anomaly have a higher weight.
- Callable: A custom function to include positional bias, which takes as input two integers (a position within the anomalous range, and the total length of that range) and returns a float (the weight of that position).
gamma (str or callable, default='reciprocal') –
Penalization approach for detecting multiple ranges with a single range. Valid options are:
- 'one': Fragmented detection should not be penalized.
- 'reciprocal': Weight fragmented detection of :math:´N´ ranges with as single range by a factor of :math:´1/N´.
- Callable: A custom function to penalize fragmented detection, which takes as input an integer (the number of detected ranges) and returns a float (the penalization factor).

class dtaianomaly.evaluation.RangeBasedRecall(alpha: float = 0.5, delta: str | Callable[[int, int], float] = 'flat', gamma: str | Callable[[int], float] = 'reciprocal')[source]

Computes the range-based recall score [24].

The range-based recall computes a recall-score for each ground truth anomalous range and then takes the average over all ranges. This recall-score consists of three parts: (1) the amount of overlap between the ground truth range and the predicted ranges, (2) whether the ground truth range overlaps with only one or multiple predicted ranges, and (3) whether the final ground truth range is detected at all. Components (1) and (2) are computed independently and multiplied, of which the result is combined with component (3) through a convex combination to get a final recall-score for the ground truth range.

Parameters:

alpha (float, default=0.5) – The importance of detecting the events (even if it is only a single detected point) compared to detecting a large portion of the ground truth events. Should be at least 0 and at most 1.
delta (str or callable, default='flat') –
Bias for the position of the predicted anomaly in the ground truth anomalous range. Valid options are:
- 'flat': Equal bias towards all positions in the ground truth anomalous range.
- 'front': Predictions that are near the front of the ground truth anomaly (i.e. early detection) have a higher weight.
- 'back': Predictions that are near the end of the ground truth anomaly (i.e. late detection) have a higher weight.
- 'middle': Predictions that are near the center of the ground truth anomaly have a higher weight.
- Callable: A custom function to include positional bias, which takes as input two integers (a position within the anomalous range, and the total length of that range) and returns a float (the weight of that position).
gamma (str or callable, default='reciprocal') –
Penalization approach for detecting multiple ranges with a single range. Valid options are:
- 'one': Fragmented detection should not be penalized.
- 'reciprocal': Weight fragmented detection of :math:´N´ ranges with as single range by a factor of :math:´1/N´.
- Callable: A custom function to penalize fragmented detection, which takes as input an integer (the number of detected ranges) and returns a float (the penalization factor).

class dtaianomaly.evaluation.RangeBasedFBeta(beta: (<class 'float'>, <class 'int'>) = 1.0, alpha: float = 0.5, delta: str | ~collections.abc.Callable[[int, int], float] = 'flat', gamma: str | ~collections.abc.Callable[[int], float] = 'reciprocal')[source]

Computes the range-based \(F_\beta\) score [24].

The range-based \(F_\beta\)-score equals the harmonic mean of the range-based precision and range-based recall. The metrics take into account three parts: (1) the amount of overlap between the ground truth ranges and the predicted ranges, (2) whether there is fragmented detection or not, and (3) whether the ground truth ranges are detected at all.

Parameters:

beta (int, float, default=1) – Desired beta parameter.
alpha (float, default=0.5) – The importance of detecting the events (even if it is only a single detected point) compared to detecting a large portion of the ground truth events. Should be at least 0 and at most 1.
delta (str or callable, default='flat') –
Bias for the position of the predicted anomaly in the ground truth anomalous range. Valid options are:
- 'flat': Equal bias towards all positions in the ground truth anomalous range.
- 'front': Predictions that are near the front of the ground truth anomaly (i.e. early detection) have a higher weight.
- 'back': Predictions that are near the end of the ground truth anomaly (i.e. late detection) have a higher weight.
- 'middle': Predictions that are near the center of the ground truth anomaly have a higher weight.
- Callable: A custom function to include positional bias, which takes as input two integers (a position within the anomalous range, and the total length of that range) and returns a float (the weight of that position).
gamma (str or callable, default='reciprocal') –
Penalization approach for detecting multiple ranges with a single range. Valid options are:
- 'one': Fragmented detection should not be penalized.
- 'reciprocal': Weight fragmented detection of :math:´N´ ranges with as single range by a factor of :math:´1/N´.
- Callable: A custom function to penalize fragmented detection, which takes as input an integer (the number of detected ranges) and returns a float (the penalization factor).