5
$\begingroup$

As far as I know, precision and recall are two single values. How we can plot a curve from these two single values? I think I should calculate a set of values for each of them but how?

afterwards, the curve can be depicted by using the fact that they are inversely related.

$$ \text{Precision}=\frac{tp}{tp+fp} $$ $$ \text{Recall}=\frac{tp}{tp+fn} $$ Where $tp = \text{True Positives}$, $fp = \text{False Positives}$ and $fn = \text{False Negatives}$.

Anybody can explain how we can plot a curve from two single values?

One answer can be found here but I do not believe that I have caught the point well.

$\endgroup$

2 Answers 2

3
$\begingroup$

You're right. When you just have a single precision and a single recall value, you get a precision-recall point, not a curve.

However, machine learning models typically do not output discrete categories, even when such models fall into the "classification" paradigm (which is why, for instance, Frank Harrell dislikes the term "classification").

For instance, a logistic regression or a neural network, using a method like model.predict_proba in Python (not just model.predict), will return values on a continuum. The model.predict method will predict the category with the highest predicted value in predict_proba, but that is a stage on top of the logistic regression or neural network itself, a stage that combines the machine learning model output with a decision rule about what to do with that output.

You get the precision-recall curve by choosing multiple decision rules, calculating the precision and recall for each decision rule, and plotting those precision-recall pairs. The multiple decision rules have to do with thresholds. As logistic regression (especially) and neural network outputs are often on the interval $[0,1]$, they can be interpreted as probabilities. The default decision rule (such as in model.predict) is to assign to category $0$ if the probability of category $1$ (which is what is predicted) is less than $0.5$, otherwise category $1$. Another decision rule might use a threshold of $0.2$ or $0.9$ instead of $0.5$.

Loop over many thresholds to produce an entire PR curve. In the code below, I let sklearn calculate the thresholds, but if you set the thresholds in for value in thresholds to be something like np.arange(0, 1, 0.001), you will get similar results.

from sklearn.metrics import precision_recall_curve, precision_score, recall_score
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(2024)

# Define some simulated model predictions and binary outcomes
#
preds = np.random.beta(1/3, 1/3, 1000)
truth = np.random.binomial(1, preds, len(preds))

# Calculate the PR curve in sklearn
#
precision, recall, thresholds = precision_recall_curve(truth, preds)

# Loop over various thresholds to calculate the PR curve ourselves
#
precision_self = []
recall_self = []
for value in thresholds:
    
    # Assign categorical predictions according to the side of the threshold on which
    # the prediction falls
    # Category 0 if < threshold
    # Category 1 otherwise
    #
    categories = [0 if item < value else 1 for item in preds]
    
    # Calculate the precision and recall scores
    #
    precision_self.append(precision_score(truth, categories))
    recall_self.append(recall_score(truth, categories))
    
# Plot it out to see that we get the same curve either way
#
plt.plot(recall, precision, label = "sklearn", linewidth = 7)
plt.plot(recall_self, precision_self, label = "self", linewidth = 3)
plt.xlabel("Recall")
plt.ylabel("Precision")
plt.legend()
plt.show()
plt.close()

sklearn vs self-implementation

A critical piece to keep in mind is that many of these machine learning models called "classifiers" do not do classification on their own. They predict on a continuum, and then some decision rule can, but does not have to and arguably should not, convert those predictions on a continuum to categories, and there are many possible decision rules, each of which has its own recall and precision values.

$\endgroup$
1
$\begingroup$

Obviously, you can't plot a curve from a single pair of values, so you have to find a parameter to vary that changes your problem in a way that makes sense to what you're trying to do.

For example, suppose your binary classification problem has a choosable threshold. If you vary the threshold up and down, then it will change the precision and recall.

Then you might get a plot like the one below. R code below that generated it.

enter image description here


R Code

#26568

Ndata <- 1000

c1 <- runif(Ndata,1.5,3)
c2 <- runif(Ndata,1,2.5)

data <- c(c1, c2)

thresholds <- seq(1,2.99,0.01)

precision <- thresholds*0
recall <- thresholds*0

for (k in 1:length(thresholds))
{
  threshold <- thresholds[k]
  cl <- data*0
  tp <- 0
  tn <- 0
  fp <- 0
  fn <- 0

  for (i in 1:length(data))
  {
    if ( (data[i] > threshold) && (i <= Ndata))
    {
      tp <- tp + 1  
    }
    if ( (data[i] > threshold) && (i > Ndata))
    {
      fp <- fp + 1  
    }
    if ( (data[i] <= threshold) && (i <= Ndata))
    {
      fn <- fn + 1  
    }
    if ( (data[i] <= threshold) && (i > Ndata))
    {
      tn <- tn + 1  
    }
  }

  precision[k] <- tp / (tp + fp)
  recall[k] <- tp / (tp  + fn)
}

plot(precision,recall)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.