Classification Metrics

ClassificationMetrics

class SeqMetrics.ClassificationMetrics(true, predicted, multiclass: bool = False, *args, **kwargs)

Bases: Metrics

Calculates classification metrics.

Parameters:

• true (array-like of shape = [n_samples] or [n_samples, n_classes]) – True class labels.

• predicted (array-like of shape = [n_samples] or [n_samples, n_classes]) – Predicted class labels.

• multiclass (boolean, optional) – If true, it is assumed that the true labels are multiclass.

• **kwargs (optional) – Additional arguments to be passed to the Metrics class.

Examples

>>> import numpy as np
>>> from SeqMetrics import ClassificationMetrics

boolean array

>>> t = np.array([True, False, False, False])
>>> p = np.array([True, True, True, True])
>>> metrics = ClassificationMetrics(t, p)
>>> accuracy = metrics.accuracy()

binary classification with numerical labels

>>> true = np.array([1, 0, 0, 0])
>>> pred = np.array([1, 1, 1, 1])
>>> metrics = ClassificationMetrics(true, pred)
>>> accuracy = metrics.accuracy()

multiclass classification with numerical labels

>>> true = np.random.randint(1, 4, 100)
>>> pred = np.random.randint(1, 4, 100)
>>> metrics = ClassificationMetrics(true, pred)
>>> accuracy = metrics.accuracy()

You can also provide logits instead of labels.

>>> predictions = np.array([[0.25, 0.25, 0.25, 0.25],
>>>                        [0.01, 0.01, 0.01, 0.96]])
>>> targets = np.array([[0, 0, 0, 1],
>>>                     [0, 0, 0, 1]])
>>> metrics = ClassificationMetrics(targets, predictions, multiclass=True)
>>> metrics.cross_entropy()
...  0.71355817782

Working with categorical values is seamless

>>> true = np.array(['a', 'b', 'b', 'b'])
>>> pred = np.array(['a', 'a', 'a', 'a'])
>>> metrics = ClassificationMetrics(true, pred)
>>> accuracy = metrics.accuracy()

same goes for multiclass categorical labels

>>> t = np.array(['car', 'truck', 'truck', 'car', 'bike', 'truck'])
>>> p = np.array(['car', 'car',   'bike',  'car', 'bike', 'truck'])
>>> metrics = ClassificationMetrics(targets, predictions, multiclass=True)
>>> print(metrics.calculate_all())

__init__(true, predicted, multiclass: bool = False, *args, **kwargs)

Parameters:

• true (array like,) – ture/observed/actual/target values

• predicted (array like,) – simulated values

• replace_nan (default None. if not None, then NaNs in true) – and predicted will be replaced by this value.

• replace_inf (default None, if not None, then inf vlaues in true and) – predicted will be replaced by this value.

• remove_zero (default False, if True, the zero values in true) – or predicted arrays will be removed. If a zero is found in one array, the corresponding value in the other array will also be removed.

• remove_neg (default False, if True, the negative values in true) – or predicted arrays will be removed.

• metric_type (type of metric.) –

• np_errstate (dict) – any keyword options for np.errstate() to calculate np.log1p

accuracy(normalize: bool = True) → float

calculates accuracy

Parameters:

normalize (bool) –

Return type:

float

Examples

>>> import numpy as np
>>> from SeqMetrics import ClassificationMetrics
>>> true = np.array([1, 0, 0, 0])
>>> pred = np.array([1, 1, 1, 1])
>>> metrics = ClassificationMetrics(true, pred)
>>> print(metrics.accuracy())

balanced_accuracy(average=None) → float

balanced accuracy. It performs better on imbalanced datasets.

confusion_matrix(normalize=False)

calculates confusion matrix

Parameters:

normalize (str, [None, 'true', 'pred', 'all]) – If None, no normalization is done.

Returns:

confusion matrix

Return type:

ndarray

Examples

>>> import numpy as np
>>> from SeqMetrics import ClassificationMetrics
>>> true = np.array([1, 0, 0, 0])
>>> pred = np.array([1, 1, 1, 1])
>>> metrics = ClassificationMetrics(true, pred)
>>> metrics.confusion_matrix()

multiclass classification

>>> true = np.random.randint(1, 4, 100)
>>> pred = np.random.randint(1, 4, 100)
>>> metrics = ClassificationMetrics(true, pred)
>>> metrics.confusion_matrix()

cross_entropy(epsilon=1e-12) → float

Computes cross entropy between targets (encoded as one-hot vectors) and predictions.

Return type:

scalar

error_rate()

Error rate is the number of all incorrect predictions divided by the total number of samples in data.

f1_score(average=None) → Union[ndarray, float]

Calculates f1 score according to following formula f1_score = 2 * (precision * recall) / (precision + recall)

Parameters:

average (str, optional) – It can be macro or weighted. or micro

Return type:

array or float

Examples

>>> import numpy as np
>>> from SeqMetrics import ClassificationMetrics
>>> true = np.array([1, 0, 0, 0])
>>> pred = np.array([1, 1, 1, 1])
>>> metrics = ClassificationMetrics(true, pred)
>>> calc_f1_score = metrics.f1_score()
...
>>> print(metrics.f1_score(average="macro"))
>>> print(metrics.f1_score(average="weighted"))

f2_score(average=None)

f2 score

false_discovery_rate()

False discovery rate

FP / (TP + FP)

false_negative_rate()

False Negative Rate or miss rate.

FN / (FN + TP)

false_omission_rate(average=None)

False omission rate

FN / (FN + TN)

false_positive_rate()

False positive rate is the number of incorrect positive predictions divided by the total number of negatives. Its best value is 0.0 and worst value is 1.0. It is also called probability of false alarm or fall-out.

TP / (TP + TN)

fowlkes_mallows_index(average=None)

Fowlkes–Mallows index

sqrt(PPV * TPR)

PPV is positive predictive value or precision. TPR is true positive rate or recall or sensitivity

https://en.wikipedia.org/wiki/Fowlkes%E2%80%93Mallows_index

mathews_corr_coeff()

Methew’s correlation coefficient

negative_likelihood_ratio(average=None)

Negative likelihood ratio

1 - sensitivity / specificity

https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing#positive_likelihood_ratio

negative_predictive_value()

Negative Predictive Value TN/(TN+FN)

positive_likelihood_ratio(average=None)

Positive likelihood ratio sensitivity / 1-specificity

precision(average=None)

Returns precision score, also called positive predictive value. It is number of correct positive predictions divided by the total number of positive predictions. TP/(TP+FP)

Parameters:

average (string, [None, macro, weighted, micro]) –

Examples

>>> import numpy as np
>>> from SeqMetrics import ClassificationMetrics
>>> true = np.array([1, 0, 0, 0])
>>> pred = np.array([1, 1, 1, 1])
>>> metrics = ClassificationMetrics(true, pred)
>>> print(metrics.precision())
...
>>> print(metrics.precision(average="macro"))
>>> print(metrics.precision(average="weighted"))

prevalence_threshold(average=None)

Prevalence threshold

sqrt(FPR) / (sqrt(TPR) + sqrt(FPR))

TPR is true positive rate or recall

recall(average=None)

It is also called sensitivity or true positive rate. It is number of correct positive predictions divided by the total number of positives Formula :

True Posivitive / True Positive + False Negative

Parameters:

average (str (default=None)) – one of None, macro, weighted, or micro

specificity(average=None)

It is also called true negative rate or selectivity. It is the probability that the predictions are negative when the true labels are also negative. It is number of correct negative predictions divided by the total number of negatives.

It’s formula is following TN / TN+FP

Examples

>>> import numpy as np
>>> from SeqMetrics import ClassificationMetrics
>>> true = np.array([1, 0, 0, 0])
>>> pred = np.array([1, 1, 1, 1])
>>> metrics = ClassificationMetrics(true, pred)
>>> print(metrics.specificity())
...
>>> print(metrics.specificity(average="macro"))
>>> print(metrics.specificity(average="weighted"))

youden_index(average=None)

Youden index, also known as informedness

j = TPR + TNR − 1 = sensitivity + specificity - 1

https://en.wikipedia.org/wiki/Youden%27s_J_statistic