# Regression Metrics

## RegressionMetrics

class SeqMetrics.RegressionMetrics(*args, **kwargs)[source]

Bases: Metrics

Calculates more than 100 regression performance metrics related to sequence data.

Example

>>> import numpy as np
>>> from SeqMetrics import RegressionMetrics
>>> t = np.random.random(10)
>>> p = np.random.random(10)
>>> errors = RegressionMetrics(t,p)
>>> all_errors = errors.calculate_all()

__init__(*args, **kwargs)[source]

Initializes Metrics.

args and kwargs go to parent class SeqMetrics.Metrics.

JS() float[source]

Jensen-shannon divergence

abs_pbias() float[source]

Absolute Percent bias

acc() float[source]

Anomaly correction coefficient. See Langland et al., 2012; Miyakoda et al., 1972 and Murphy et al., 1989.

agreement_index() float[source]

Agreement Index (d) developed by Willmott, 1981.

It detects additive and pro-portional differences in the observed and simulated means and vari-ances Moriasi et al., 2015. It is overly sensitive to extreme values due to the squared differences. It can also be used as a substitute for R2 to identify the degree to which model predic-tions are error-free.

$d = 1 - \frac{\sum_{i=1}^{N}(e_{i} - s_{i})^2}{\sum_{i=1}^{N}(\left | s_{i} - \bar{e} \right | + \left | e_{i} - \bar{e} \right |)^2}$
aic(p=1) float[source]

Akaike Information Criterion. Modifying from this source

aitchison(center='mean') float[source]

Aitchison distance. used in Zhang et al., 2020

amemiya_pred_criterion() float[source]

Amemiya’s Prediction Criterion

bias() float[source]

Bias as and given by Gupta1998 et al., 1998

$Bias=\frac{1}{N}\sum_{i=1}^{N}(e_{i}-s_{i})$
bic(p=1) float[source]

Bayesian Information Criterion

Minimising the BIC is intended to give the best model. The model chosen by the BIC is either the same as that chosen by the AIC, or one with fewer terms. This is because the BIC penalises the number of parameters more heavily than the AIC. Modified after RegscorePy.

brier_score() float[source]

Adopted from SkillMetrics Calculates the Brier score (BS), a measure of the mean-square error of probability forecasts for a dichotomous (two-category) event, such as the occurrence/non-occurrence of precipitation. The score is calculated using the formula:

$BS = sum_(n=1)^N (f_n - o_n)^2/N$

where f is the forecast probabilities, o is the observed probabilities (0 or 1), and N is the total number of values in f & o. Note that f & o must have the same number of values, and those values must be in the range [0,1].

Returns

BS : Brier score

Return type

float

References

Glenn W. Brier, 1950: Verification of forecasts expressed in terms of probabilities. Mon. We. Rev., 78, 1-23. D. S. Wilks, 1995: Statistical Methods in the Atmospheric Sciences. Cambridge Press. 547 pp.

calculate_hydro_metrics()[source]

Calculates all metrics for hydrological data.

Returns

Dictionary with all metrics

Return type

dict

centered_rms_dev() float[source]

Modified after SkillMetrics. Calculates the centered root-mean-square (RMS) difference between true and predicted using the formula: (E’)^2 = sum_(n=1)^N [(p_n - mean(p))(r_n - mean(r))]^2/N where p is the predicted values, r is the true values, and N is the total number of values in p & r.

Output: CRMSDIFF : centered root-mean-square (RMS) difference (E’)^2

corr_coeff() float[source]

Pearson correlation coefficient. It measures linear correlatin between true and predicted arrays. It is sensitive to outliers. Reference: Pearson, K 1895.

$r = \frac{\sum ^n _{i=1}(e_i - \bar{e})(s_i - \bar{s})}{\sqrt{\sum ^n _{i=1}(e_i - \bar{e})^2} \sqrt{\sum ^n _{i=1}(s_i - \bar{s})^2}}$
cosine_similarity() float[source]

It is a judgment of orientation and not magnitude: two vectors with the same orientation have a cosine similarity of 1, two vectors oriented at 90° relative to each other have a similarity of 0, and two vectors diametrically opposed have a similarity of -1, independent of their magnitude. See

covariance() float[source]
Covariance


Covariance = frac{1}{N} sum_{i=1}^{N}((e_{i} - bar{e}) * (s_{i} - bar{s}))

cronbach_alpha() float[source]

It is a measure of internal consitency of data. See ucla and stackoverflow pages for more info.

decomposed_mse() float[source]

Decomposed MSE developed by Kobayashi and Salam (2000)

$dMSE = (\frac{1}{N}\sum_{i=1}^{N}(e_{i}-s_{i}))^2 + SDSD + LCS SDSD = (\sigma(e) - \sigma(s))^2 LCS = 2 \sigma(e) \sigma(s) * (1 - \frac{\sum ^n _{i=1}(e_i - \bar{e})(s_i - \bar{s})} {\sqrt{\sum ^n _{i=1}(e_i - \bar{e})^2} \sqrt{\sum ^n _{i=1}(s_i - \bar{s})^2}})$
euclid_distance() float[source]

Euclidian distance

Referneces: Kennard et al., 2010

exp_var_score(weights=None) Optional[float][source]

Explained variance score . Best value is 1, lower values are less accurate.

expanded_uncertainty(cov_fact=1.96) float[source]

By default it calculates uncertainty with 95% confidence interval. 1.96 is the coverage factor corresponding 95% confidence level .This indicator is used in order to show more information about the model deviation. Using formula from by Behar et al., 2015 and Gueymard et al., 2014.

fdc_fhv(h: float = 0.02) float[source]

modified Kratzert2018 code. Peak flow bias of the flow duration curve (Yilmaz 2008). used in kratzert et al., 2018

Parameters

h (float) – Must be between 0 and 1.

Return type

Bias of the peak flows

fdc_flv(low_flow: float = 0.3) float[source]

bias of the bottom 30 % low flows. modified Kratzert code used in kratzert et al., 2018

Parameters

low_flow (float, optional) – Upper limit of the flow duration curve. E.g. 0.3 means the bottom 30% of the flows are considered as low flows, by default 0.3

Return type

float

gmae() float[source]

Geometric Mean Absolute Error

gmean_diff() float[source]

Geometric mean difference. First geometric mean is calculated for each of two samples and their difference is calculated.

gmrae(benchmark: Optional[ndarray] = None) float[source]

Geometric Mean Relative Absolute Error

inrse() float[source]

Integral Normalized Root Squared Error

irmse() float[source]

Inertial RMSE. RMSE divided by standard deviation of the gradient of true.

kendaull_tau(return_p=False) Union[float, tuple][source]

Kendall’s tau .used in Probst et al., 2019.

kge(return_all=False)[source]

Kling-Gupta Efficiency Gupta, Kling, Yilmaz, Martinez, 2009, Decomposition of the mean squared error and NSE performance

criteria: Implications for improving hydrological modelling

output:

kge: Kling-Gupta Efficiency cc: correlation alpha: ratio of the standard deviation beta: ratio of the mean

kge_bound() float[source]

Bounded Version of the Original Kling-Gupta Efficiency_

kge_mod(return_all=False)[source]

Modified Kling-Gupta Efficiency_ .

kge_np(return_all=False)[source]

Non parametric Kling-Gupta Efficiency

output:

kge: Kling-Gupta Efficiency cc: correlation alpha: ratio of the standard deviation beta: ratio of the mean

References

Pool, Vis, and Seibert, 2018 Evaluating model performance: towards a non-parametric variant of the Kling-Gupta efficiency, Hydrological Sciences Journal. https://doi.org/10.1080/02626667.2018.1552002

kgenp_bound()[source]

Bounded Version of the Non-Parametric Kling-Gupta Efficiency

kgeprime_c2m() float[source]

Bounded Version of the Modified Kling-Gupta Efficiency_

kl_sym() Optional[float][source]

Symmetric kullback-leibler divergence

lm_index(obs_bar_p=None) float[source]

Legate-McCabe Efficiency Index. Less sensitive to outliers in the data. The larger, the better

Parameters

obs_bar_p (float,) – Seasonal or other selected average. If None, the mean of the observed array will be used.

log_nse(epsilon=0.0) float[source]

log Nash-Sutcliffe model efficiency

$NSE = 1-\frac{\sum_{i=1}^{N}(log(e_{i})-log(s_{i}))^2}{\sum_{i=1}^{N}(log(e_{i})-log(\bar{e})^2}-1)*-1$
log_prob() float[source]

Logarithmic probability distribution

maape() float[source]

Mean Arctangent Absolute Percentage Error Note: result is NOT multiplied by 100

mae(true=None, predicted=None) float[source]

Mean Absolute Error. It is less sensitive to outliers as compared to mse/rmse.

mapd() float[source]

Mean absolute percentage deviation.

mape() float[source]

Mean Absolute Percentage Error. The MAPE is often used when the quantity to predict is known to remain way above zero. It is useful when the size or size of a prediction variable is significant in evaluating the accuracy of a prediction. It has advantages of scale-independency and interpretability. However, it has the significant disadvantage that it produces infinite or undefined values for zero or close-to-zero actual values.

mare() float[source]

Mean Absolute Relative Error. When expressed in %age, it is also known as mape.

mase(seasonality: int = 1)[source]

Mean Absolute Scaled Error. Baseline (benchmark) is computed with naive forecasting (shifted by @seasonality) modified after 11. It is the ratio of MAE of used model and MAE of naive forecast.

References

11

https://gist.github.com/bshishov/5dc237f59f019b26145648e2124ca1c9

Hyndman, R. J. (2006). Another look at forecast-accuracy metrics for intermittent demand. Foresight: The International Journal of Applied Forecasting, 4(4), 43-46.

max_error() float[source]

maximum absolute error

mb_r() float[source]

Mielke-Berry R value. Berry and Mielke, 1988.

References

Mielke, P. W., & Berry, K. J. (2007). Permutation methods: a distance function approach. Springer Science & Business Media.

mbe() float[source]

Mean bias error. This indicator expresses a tendency of model to underestimate (negative value) or overestimate (positive value) global radiation, while the MBE values closest to zero are desirable. The drawback of this test is that it does not show the correct performance when the model presents overestimated and underestimated values at the same time, since overestimation and underestimation values cancel each other. [1]

mbrae(benchmark: Optional[ndarray] = None) float[source]

Mean Bounded Relative Absolute Error

mda() float[source]

Mean Directional Accuracy modified after

mdape() float[source]

Median Absolute Percentage Error

mde() float[source]

Median Error

mdrae(benchmark: Optional[ndarray] = None) float[source]

Median Relative Absolute Error

me()[source]

Mean error

mean_bias_error() float[source]

Mean Bias Error It represents overall bias error or systematic error. It shows average interpolation bias; i.e. average over- or underestimation. [1][2].This indicator expresses a tendency of model to underestimate (negative value) or overestimate (positive value) global radiation, while the MBE values closest to zero are desirable. The drawback of this test is that it does not show the correct performance when the model presents overestimated and underestimated values at the same time, since overestimation and underestimation values cancel each other.

References

mean_gamma_deviance(weights=None) float[source]

mean gamma deviance

mean_poisson_deviance(weights=None) float[source]

mean poisson deviance

mean_var() float[source]

Mean variance

med_seq_error() float[source]

Median Squared Error Same as mse but it takes median which reduces the impact of outliers.

median_abs_error() float[source]

median absolute error

mle() float[source]

Mean log error

mod_agreement_index(j=1) float[source]

Modified agreement of index. j: int, when j==1, this is same as agreement_index. Higher j means more impact of outliers.

mpe() float[source]

Mean Percentage Error

mrae(benchmark: Optional[ndarray] = None)[source]

Mean Relative Absolute Error

msle(weights=None) float[source]

mean square logrithmic error

norm_ae() float[source]

Normalized Absolute Error

norm_ape() float[source]

Normalized Absolute Percentage Error

norm_euclid_distance() float[source]

Normalized Euclidian distance

nrmse() float[source]

Normalized Root Mean Squared Error

nrmse_ipercentile(q1=25, q2=75) float[source]

RMSE normalized by inter percentile range of true. This is least sensitive to outliers. q1: any interger between 1 and 99 q2: any integer between 2 and 100. Should be greater than q1. Reference: Pontius et al., 2008.

nrmse_mean() float[source]

Mean Normalized RMSE RMSE normalized by mean of true values.This allows comparison between datasets with different scales.

Reference: Pontius et al., 2008

nrmse_range() float[source]

Range Normalized Root Mean Squared Error. RMSE normalized by true values. This allows comparison between data sets with different scales. It is more sensitive to outliers.

Reference: Pontius et al., 2008

nse() float[source]

Nash-Sutcliff Efficiency.

It determine how well the model simulates trends for the output response of concern. But cannot help identify model bias and cannot be used to identify differences in timing and magnitude of peak flows and shape of recession curves; in other words, it cannot be used for single-event simulations. It is sensitive to extreme values due to the squared differ-ences [1]. To make it less sensitive to outliers, [2] proposed log and relative nse.

References

• Moriasi, D. N., Gitau, M. W., Pai, N., & Daggupati, P. (2015). Hydrologic and water quality models:

Performance measures and evaluation criteria. Transactions of the ASABE, 58(6), 1763-1785.

• Krause, P., Boyle, D., & Bäse, F. (2005). Comparison of different efficiency criteria for hydrological

nse_alpha() float[source]

Alpha decomposition of the NSE, see Gupta_ et al. 2009 used in kratzert et al., 2018

Returns

Alpha decomposition of the NSE

Return type

float

nse_beta() float[source]

Beta decomposition of NSE. See Gupta et. al 2009 . used in kratzert et al., 2018

Returns

Beta decomposition of the NSE

Return type

float

nse_bound() float[source]

Bounded Version of the Nash-Sutcliffe Efficiency (nse)

nse_mod(j=1) float[source]

Gives less weightage of outliers if j=1 and if j>1, gives more weightage to outliers. Reference: Krause et al., 2005

nse_rel() float[source]

Relative NSE.

pbias() float[source]

Percent Bias. It determine how well the model simulates the average magnitudes for the output response of interest. It can also determine over and under-prediction. It cannot be used (1) for single-event simula-tions to identify differences in timing and magnitude of peak flows and the shape of recession curves nor (2) to determine how well the model simulates residual variations and/or trends for the output response of interest. It can give a deceiving rating of model performance if the model overpredicts as much as it underpredicts, in which case PBIAS will be close to zero even though the model simulation is poor. [1]

[1] Moriasi et al., 2015

r2() float[source]

Quantifies the percent of variation in the response that the ‘model’ explains. The ‘model’ here is anything from which we obtained predicted array. It is also called coefficient of determination or square of pearson correlation coefficient. More heavily affected by outliers than pearson correlatin r.

r2_score(weights=None)[source]

This is not a symmetric function. Unlike most other scores, R^2 score may be negative (it need not actually be the square of a quantity R). This metric is not well-defined for single samples and will return a NaN value if n_samples is less than two.

rae() float[source]

Relative Absolute Error (aka Approximation Error)

ref_agreement_index() float[source]

Refined Index of Agreement. From -1 to 1. Larger the better. Refrence: Willmott et al., 2012

rel_agreement_index() float[source]

Relative index of agreement. from 0 to 1. larger the better.

relative_rmse() float[source]

Relative Root Mean Squared Error

$RRMSE=\frac{\sqrt{\frac{1}{N}\sum_{i=1}^{N}(e_{i}-s_{i})^2}}{\bar{e}}$
rmdspe() float[source]

Root Median Squared Percentage Error

rmse(weights=None) float[source]

root mean square error

rmsle() float[source]

Root mean square log error.

This error is less sensitive to outliers . Compared to RMSE, RMSLE only considers the relative error between predicted and actual values, and the scale of the error is nullified by the log-transformation. Furthermore, RMSLE penalizes underestimation more than overestimation. This is especially useful in those studies where the underestimation of the target variable is not acceptable but overestimation can be tolerated .

rmspe() float[source]

Root Mean Square Percentage Error_ .

rmsse(seasonality: int = 1) float[source]

Root Mean Squared Scaled Error

rrse() float[source]

Root Relative Squared Error

rse() float[source]

Relative Squared Error

rsr() float[source]

Moriasi et al., 2007. It incorporates the benefits of error index statistics andincludes a scaling/normalization factor, so that the resulting statistic and reported values can apply to various constitu-ents.

sa() float[source]

Spectral angle. From -pi/2 to pi/2. Closer to 0 is better. It measures angle between two vectors in hyperspace indicating how well the shape of two arrays match instead of their magnitude. Reference: Robila and Gershman, 2005.

sc() float[source]

Spectral correlation. It varies from -pi/2 to pi/2. Closer to 0 is better.

sga() float[source]

Spectral gradient angle. It varies from -pi/2 to pi/2. Closer to 0 is better.

sid() float[source]

Spectral Information Divergence. From -pi/2 to pi/2. Closer to 0 is better.

skill_score_murphy() float[source]

Adopted from here . Calculate non-dimensional skill score (SS) between two variables using definition of Murphy (1988) using the formula:

\begin{align}\begin{aligned}SS = 1 - RMSE^2/SDEV^2\\SDEV is the standard deviation of the true values\\SDEV^2 = sum_(n=1)^N [r_n - mean(r)]^2/(N-1)\end{aligned}\end{align}

where p is the predicted values, r is the reference values, and N is the total number of values in p & r. Note that p & r must have the same number of values. A positive skill score can be interpreted as the percentage of improvement of the new model forecast in comparison to the reference. On the other hand, a negative skill score denotes that the forecast of interest is worse than the referencing forecast. Consequently, a value of zero denotes that both forecasts perform equally [MLAir, 2020].

Returns

flaot

References

Allan H. Murphy, 1988: Skill Scores Based on the Mean Square Error and Their Relationships to the Correlation Coefficient. Mon. Wea. Rev., 116, 2417-2424. doi: http//dx.doi.org/10.1175/1520-0493(1988)<2417:SSBOTM>2.0.CO;2

smape() float[source]

Symmetric Mean Absolute Percentage Error_. Adoption from .

smdape() float[source]

Symmetric Median Absolute Percentage Error Note: result is NOT multiplied by 100

spearmann_corr() float[source]

Separmann correlation coefficient.

This is a nonparametric metric and assesses how well the relationship between the true and predicted data can be described using a monotonic function.

sse() float[source]

Sum of squared errors_ (model vs actual). It is measure of how far off our model’s predictions are from the observed values. A value of 0 indicates that all predications are spot on. A non-zero value indicates errors.

This is also called residual sum of squares (RSS) or sum of squared residuals as per tutorialspoint .

std_ratio(**kwargs) float[source]

ratio of standard deviations of predictions and trues. Also known as standard ratio, it varies from 0.0 to infinity while 1.0 being the perfect value.

umbrae(benchmark: Optional[ndarray] = None)[source]

Unscaled Mean Bounded Relative Absolute Error

ve() float[source]

Volumetric efficiency. from 0 to 1. Smaller the better. Reference: Criss and Winston 2008.

volume_error() float[source]

Returns the Volume Error (Ve). It is an indicator of the agreement between the averages of the simulated and observed runoff (i.e. long-term water balance). used in Reynolds paper:

$Sum(self.predicted- true)/sum(self.predicted)$

References

Reynolds, J.E., S. Halldin, C.Y. Xu, J. Seibert, and A. Kauffeldt. 2017. “Sub-Daily Runoff Predictions Using Parameters Calibrated on the Basis of Data with a Daily Temporal Resolution.” Journal of Hydrology 550 (July):399?411.

wape() float[source]

weighted absolute percentage error (wape)

It is a variation of mape but more suitable for intermittent and low-volume data.

watt_m() float[source]

Watterson’s M. Refrence: Watterson., 1996

wmape() float[source]

Weighted Mean Absolute Percent Error_ .