A Brief Overview of Multiple Hypothesis Testing
By Nathan Nguyen
May 9, 2024
Simulating the Family Wise Error Rate (FWER) for various \(\alpha\)
-levels
The code below is a simulation of the FWER for various \(\alpha\)
-levels and 500 hundred hypothesis tests.
# define number of hypothesis tests
num_tests <- 1:500
# define alpha values
alphas <- c(0.001, 0.01, 0.05)
# define fwer function
fwer_function <- function(num_tests, alpha) {
1 - (1 - alpha)^num_tests
}
fwer_data <- expand.grid(num_tests = num_tests, alpha = alphas) %>%
mutate(fwer = fwer_function(num_tests, alpha))
fwer_data %>%
ggplot(aes(x = num_tests, y = fwer, color = factor(alpha), group = alpha)) +
geom_line() +
scale_color_lancet() +
# scale_x_log10() +
labs(title = "Family-wise Error Rate for Different Significance Levels",
subtitle = "Simulated for 500 Hypothesis Tests",
x = "Number of Hypotheses",
y = "Family-Wise Error Rate",
color = expression(alpha-level),
caption = "Figure inspired by An Introduction to Statistical Learning") +
theme_bw() +
geom_hline(yintercept = 0.05, linetype = "dashed", color = "black") +
scale_x_log10(breaks = c(1, 5, 10, 50, 100, 500),
labels = c("1", "5", "10", "50", "100", "500"))
$$
\begin{aligned} FWER &= 1 - (1-\alpha)^{m} \end{aligned}
$$
Where \(m\)
is the total number of tests performed, and \(\alpha\)
is the significance level.
reactable(
fwer_data,
columns = list(
num_tests = colDef(name = "Number of Hypothesis Tests"),
alpha = colDef(name = "\u03B1-level"), # unicode for the alpha character
fwer = colDef(name = "Family-Wise Error Rate")
),
filterable = TRUE
)
- Posted on:
- May 9, 2024
- Length:
- 51 minute read, 10747 words
- See Also: