A Brief Overview of Multiple Hypothesis Testing

By Nathan Nguyen

May 9, 2024

Simulating the Family Wise Error Rate (FWER) for various α\alpha-levels

The code below is a simulation of the FWER for various α\alpha-levels and 500 hundred hypothesis tests.

# define number of hypothesis tests
num_tests <- 1:500

# define alpha values
alphas <- c(0.001, 0.01, 0.05)

# define fwer function
fwer_function <- function(num_tests, alpha) {
  1 - (1 - alpha)^num_tests
}

fwer_data <- expand.grid(num_tests = num_tests, alpha = alphas) %>% 
  mutate(fwer = fwer_function(num_tests, alpha))

fwer_data %>% 
  ggplot(aes(x = num_tests, y = fwer, color = factor(alpha), group = alpha)) +
  geom_line() +
  scale_color_lancet() +
  # scale_x_log10() +
  labs(title = "Family-wise Error Rate for Different Significance Levels",
       subtitle = "Simulated for 500 Hypothesis Tests",
       x = "Number of Hypotheses",
       y = "Family-Wise Error Rate",
       color = expression(alpha-level),
       caption = "Figure inspired by An Introduction to Statistical Learning") +
  theme_bw() +
  geom_hline(yintercept = 0.05, linetype = "dashed", color = "black") +
  scale_x_log10(breaks = c(1, 5, 10, 50, 100, 500),
                labels = c("1", "5", "10", "50", "100", "500"))
1510501005000.000.250.500.751.00
expression(alpha - level)0.0010.010.05Family-wise Error Rate for Different Significance LevelsNumber of HypothesesFamily-Wise Error Rate

FWER=1(1α)m \begin{aligned} FWER &= 1 - (1-\alpha)^{m} \end{aligned}

Where mm is the total number of tests performed, and α\alpha is the significance level.

reactable(
  fwer_data,
  columns = list(
    num_tests = colDef(name = "Number of Hypothesis Tests"),
    alpha = colDef(name = "\u03B1-level"), # unicode for the alpha character
    fwer = colDef(name = "Family-Wise Error Rate")
  ),
  filterable = TRUE
)
Number of Hypothesis Tests
α-level
Family-Wise Error Rate
1
0.001
0.001
2
0.001
0.00199899999999997
3
0.001
0.002997001
4
0.001
0.00399400399899996
5
0.001
0.00499000999500099
6
0.001
0.005985019985006
7
0.001
0.00697903496502095
8
0.001
0.00797205593005601
9
0.001
0.00896408387412595
10
0.001
0.00995511979025177
1–10 of 1500 rows
...
Posted on:
May 9, 2024
Length:
51 minute read, 10747 words
See Also: