---
title: "Metric types"
author: "Davis Vaughan"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Metric types}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  chunk_output_type: console
---

```{r}
#| label: setup
#| include: false
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Metric types

There are different main metric types in `yardstick`: class, class probability, 
numeric, and survival. Each type of metric has standardized argument syntax, and all metrics 
return the same kind of output (a tibble with 3 columns). This standardization 
allows metrics to easily be grouped together and used with grouped data frames 
for computing on multiple resamples at once. Below are the different types of 
metrics, along with the types of the inputs they take.

1) **Numeric metrics**

    - `truth` - numeric
    
    - `estimate` - numeric

2) **Class metrics** (hard predictions)

    - `truth` - factor
    
    - `estimate` - factor
  
3) **Class probability metrics** (soft predictions)
    - `truth` - factor
    
    - `estimate / ...` - multiple numeric columns containing class probabilities
  
3) **Ordered probability metrics** (soft predictions)
    - `truth` - ordered factor
    
    - `estimate / ...` - multiple numeric columns containing class probabilities
  
5) **Static survival metrics**

    - `truth` - Surv

    - `estimate` - numeric

6) **Dynamic survival metrics** (one value per evaluation time)

    - `truth` - Surv

    - `...` - list of data.frames, each containing the 3 columns `.eval_time`, `.pred_survival`, and `.weight_censored`

7) **Integrated survival metrics** (one value across evaluation times)

    - `truth` - Surv

    - `...` - list of data.frames, each containing the 3 columns `.eval_time`, `.pred_survival`, and `.weight_censored`

8) **Linear predictor survival metrics**

    - `truth` - Surv

    - `estimate` - numeric

9) **Quantile metrics**

    - `truth` - numeric

    - `estimate` - quantile_pred

## Example

In the following example, the `hpc_cv` data set is used. It contains class
probabilities and class predictions for a linear discriminant analysis fit to
the HPC data set of Kuhn and Johnson (2013). It is fit with 10 fold cross-validation,
and the predictions for all folds are included.

```{r}
#| warning: false
#| message: false
library(yardstick)
library(dplyr)
data("hpc_cv")

hpc_cv |>
  group_by(Resample) |>
  slice(1:3)
```

1 metric, 1 resample

```{r}
hpc_cv |>
  filter(Resample == "Fold01") |>
  accuracy(obs, pred)
```

1 metric, 10 resamples

```{r}
hpc_cv |>
  group_by(Resample) |>
  accuracy(obs, pred)
```

2 metrics, 10 resamples

```{r}
class_metrics <- metric_set(accuracy, kap)

hpc_cv |>
  group_by(Resample) |>
  class_metrics(obs, estimate = pred)
```

## Metrics

Below is a table of all of the metrics available in `yardstick`, grouped
by type.

```{r}
#| echo: false
#| warning: false
#| message: false
#| results: asis
library(knitr)
library(dplyr)

yardns <- asNamespace("yardstick")
fns <- lapply(names(yardns), get, envir = yardns)
names(fns) <- names(yardns)

get_metrics <- function(fns, type) {
  where <- vapply(fns, inherits, what = type, FUN.VALUE = logical(1))
  paste0("`", sort(names(fns[where])), "()`")
}

all_metrics <- bind_rows(
  tibble(type = "class", metric = get_metrics(fns, "class_metric")),
  tibble(type = "class prob", metric = get_metrics(fns, "prob_metric")),
  tibble(
    type = "ordered prob",
    metric = get_metrics(fns, "ordered_prob_metric")
  ),
  tibble(type = "numeric", metric = get_metrics(fns, "numeric_metric")),
  tibble(
    type = "dynamic survival",
    metric = get_metrics(fns, "dynamic_survival_metric")
  ),
  tibble(
    type = "integrated survival",
    metric = get_metrics(fns, "integrated_survival_metric")
  ),
  tibble(
    type = "static survival",
    metric = get_metrics(fns, "static_survival_metric")
  ),
  tibble(
    type = "linear predictor survival",
    metric = get_metrics(fns, "linear_pred_survival_metric")
  ),
  tibble(
    type = "quantile",
    metric = get_metrics(fns, "quantile_metric")
  )
)

kable(all_metrics, format = "html")
```