A Brief Overview of Multiple Hypothesis Testing

By Nathan Nguyen

May 9, 2024

Simulating the Family Wise Error Rate (FWER) for various \(\alpha\)-levels

The code below is a simulation of the FWER for various \(\alpha\)-levels and 500 hundred hypothesis tests.

# define number of hypothesis tests
num_tests <- 1:500

# define alpha values
alphas <- c(0.001, 0.01, 0.05)

# define fwer function
fwer_function <- function(num_tests, alpha) {
  1 - (1 - alpha)^num_tests
}

fwer_data <- expand.grid(num_tests = num_tests, alpha = alphas) %>% 
  mutate(fwer = fwer_function(num_tests, alpha))

fwer_data %>% 
  ggplot(aes(x = num_tests, y = fwer, color = factor(alpha), group = alpha)) +
  geom_line() +
  scale_color_lancet() +
  # scale_x_log10() +
  labs(title = "Family-wise Error Rate for Different Significance Levels",
       subtitle = "Simulated for 500 Hypothesis Tests",
       x = "Number of Hypotheses",
       y = "Family-Wise Error Rate",
       color = expression(alpha-level),
       caption = "Figure inspired by An Introduction to Statistical Learning") +
  theme_bw() +
  geom_hline(yintercept = 0.05, linetype = "dashed", color = "black") +
  scale_x_log10(breaks = c(1, 5, 10, 50, 100, 500),
                labels = c("1", "5", "10", "50", "100", "500"))

$$ \begin{aligned} FWER &= 1 - (1-\alpha)^{m} \end{aligned} $$

Where \(m\) is the total number of tests performed, and \(\alpha\) is the significance level.

reactable(
  fwer_data,
  columns = list(
    num_tests = colDef(name = "Number of Hypothesis Tests"),
    alpha = colDef(name = "\u03B1-level"), # unicode for the alpha character
    fwer = colDef(name = "Family-Wise Error Rate")
  ),
  filterable = TRUE
)
Posted on:
May 9, 2024
Length:
51 minute read, 10747 words
See Also: