25, 26, 27, 42
In this exercise, we consider how the power of a hypothesis test (the ability to reject the null hypothesis when it is in fact false) depends on the size of the rejection region / significance level of the test, the true difference of the population mean from the null hypothesis mean, and the sample size.
Consider the case of testing a claim about the mean from a normal population. Call the value of the mean of the population \(\mu\), and the value under the null hypothesis \(\mu_{0}\). We wish to perform the hypothesis test \[\begin{aligned} H_{0} &: \mu \leq \mu_{0} \\ H_{1} &: \mu > \mu_{0} \end{aligned}\] We will use the \(Z\) statistic \[\begin{aligned} Z = \frac{\bar{X} - \mu_{0}}{\sigma/\sqrt{n}}\end{aligned}\] as our test statistic. We know that under the null hypothesis that \(\mu = \mu_{0}\), so under the null hypothesis \(Z\) will have a standard normal distribution with mean 0 and standard deviation 1.
Suppose instead that the true mean of the population is not \(\mu_{0}\), but \(\mu_{1} > \mu_{0}\). That is, the alternative hypothesis is true, since the population mean is in fact larger than the null value \(\mu_{0}\). What is the distribution of \(Z\) now? If \(X\) is normal with mean \(\mu_{1}\) and standard deviation \(\sigma\), then \(\bar{X}\) is normal with mean \(\mu_{1}\) and standard deviation \(\frac{\sigma}{\sqrt{n}}\). So \[\begin{aligned} \frac{\bar{X} - \mu_{1}}{\sigma/\sqrt{n}}\end{aligned}\] is standard normal. But, the \(Z\) statistic that we compute is not standard normal, since we are subtracting off \(\mu_{0}\) instead of \(\mu_{1}\). In fact, we see that \[\begin{aligned} Z &= \frac{\bar{X} - \mu_{0}}{\sigma/\sqrt{n}} \\ &= \frac{\bar{X} - \mu_{0} + \mu_{1} - \mu_{1}}{\sigma/\sqrt{n}} \\ &= \frac{\bar{X} - \mu_{1}}{\sigma/\sqrt{n}} + \frac{\mu_{1} - \mu_{0}}{\sigma/\sqrt{n}}\end{aligned}\] is nonstandard normal with mean \(\dfrac{\mu_{1} - \mu_{0}}{\sigma/\sqrt{n}} = \sqrt{n}\cdot\dfrac{\mu_{1} - \mu_{0}}{\sigma}\) and standard deviation 1. (The first term in the sum is standard normal, and the second term shifts the mean of the test statistic to the right by \(\sqrt{n}\cdot\dfrac{\mu_{1} - \mu_{0}}{\sigma}\).)
The term \(\Delta = \dfrac{\mu_{1} - \mu_{0}}{\sigma}\) is called the effect size of the difference from the null hypothesis: it measures how much the true mean differs from the null mean in units of the standard deviation of the population.
Using the demo here, use the right tailed-test to explore the trade-off between the size of the rejection region and the power of the test as a function of the effect size and the sample size:
Using an effect size of 1 and a sample size of 1, determine the power of the test to reject the null hypothesis when the size of the rejection region \(\alpha\) is set to: 0.01, 0.05, 0.1, 0.5. What happens to the power of the test as you increase the size of the rejection region?
Increase the effect size to 2, keeping the sample size at 1, and repeat the power determination from the previous item. How did increasing the effect size change your answers from the previous item?
Fixing the effect size at 1 and \(\alpha\) to 0.05, determine the power of the test to reject the null hypothesis when the sample size \(n\) is set to: 1, 10, 20, 50, 100. What happens to the power of the test as the sample size increases?
Repeat the previous item, using an effect size of 0.1. How does the smaller effect size change the power of the test to reject the null hypothesis with increasing sample size?