Type I and Type II Errors

A2 · S2 · 16 min

A hypothesis test makes a decision from incomplete information, so it can be wrong. It can reject a null hypothesis that is actually true, or it can fail to reject one that is actually false. These two mistakes are called Type I and Type II errors. Paper 6 asks you to describe them in context, to say which one could have been made in a given test, and to calculate their probabilities for binomial, Poisson and normal tests. A Type I and Type II part ends a large share of the hypothesis-testing questions on the paper, and it is often where the hardest marks are.

Two ways to be wrong

A test has two possible decisions, and the truth has two possible states, so there are four combinations.

	$H_0$ is true	$H_0$ is false
Reject $H_0$	Type I error	Correct decision
Do not reject $H_0$	Correct decision	Type II error

Definition

A Type I error is rejecting $H_0$ when $H_0$ is true.
A Type II error is not rejecting (accepting) $H_0$ when $H_0$ is false.

A courtroom analogy helps: convicting an innocent person is a Type I error; acquitting a guilty one is a Type II error.

Because each error belongs to one decision, only one of them can have happened in any particular test. If $H_0$ was rejected, the only possible error is Type I. If $H_0$ was not rejected, the only possible error is Type II. This is a favourite one-mark question.

The probability of a Type I error

A Type I error happens when $H_0$ is true but the test statistic falls in the critical region. So

P(\text{Type I error}) = P(\text{test statistic in the critical region} \mid H_0 \text{ true}).

For a test using a normal distribution (a test for a mean), the critical region is chosen to have probability exactly $\alpha$ , so $P(\text{Type I error})$ equals the significance level.
For a test using binomial or Poisson probabilities, the critical region usually has probability less than $\alpha$ . Then $P(\text{Type I error})$ is the actual significance level: the exact probability of the critical region under $H_0$ . You must find the critical region first.

The probability of a Type II error

A Type II error happens when $H_0$ is false but the test statistic falls in the acceptance region. To calculate a probability you need to know the true value of the parameter, so a question will always give one: "find the probability of a Type II error if the true proportion is $0.1$ ".

P(\text{Type II error}) = P(\text{test statistic not in the critical region} \mid \text{the given true value}).

The critical region is still the one found using $H_0$ . Only the distribution used to calculate the probability changes.

Probabilities of Type I and Type II errors

Find the critical region using the distribution under $H_0$ (this may already have been done in an earlier part).
$P(\text{Type I error})$ : the probability of the critical region under $H_0$ . For a normal test this is just $\alpha$ .
For a Type II error, write down the distribution of the test statistic with the true parameter value given in the question.
$P(\text{Type II error})$ : the probability, under that true distribution, that the test statistic is not in the critical region (it lies in the acceptance region).

The graph shows both errors for a test of $H_0: \mu = 1000$ against $H_1: \mu < 1000$ , using the mean of $16$ bags with $\sigma = 8$ g, so that $\bar{X}$ has standard deviation $2$ . The right-hand curve is the distribution of $\bar{X}$ if $H_0$ is true; the left-hand curve is its distribution if in fact $\mu = 995$ . The critical region is $\bar{x} < 996.71$ . The shaded area on the left, under the $H_0$ curve, is $P(\text{Type I error}) = 0.05$ . The shaded area on the right, under the $\mu = 995$ curve, is $P(\text{Type II error})$ .

y = exp(-(x - 1000)^2 / 8) / (2 sqrt(2 pi)) y = exp(-(x - 995)^2 / 8) / (2 sqrt(2 pi)) fill 988 996.71 y = exp(-(x - 1000)^2 / 8) / (2 sqrt(2 pi)) fill 996.71 1003 y = exp(-(x - 995)^2 / 8) / (2 sqrt(2 pi))

The picture shows the trade-off between the two errors. Moving the critical value to the left (a smaller significance level) shrinks the Type I area but enlarges the Type II area. The only way to reduce both at once is to make both curves narrower, by taking a larger sample.

Routine: a binomial test

It is claimed that $30\%$ of customers at a shop use a discount code. The manager suspects the proportion is lower and tests the claim at the $5\%$ significance level, using a random sample of $20$ customers.

(a) Find the critical region and the probability of a Type I error.

(b) Find the probability of a Type II error if the true proportion is $0.1$ .

Solution

(a) $H_0: p = 0.3$ , $\quad H_1: p < 0.3$ . Under $H_0$ , $X \sim B(20, 0.3)$ .

P(X \le 2) = 0.0355 \le 0.05, \qquad P(X \le 3) = 0.1071 > 0.05.

The critical region is $X \le 2$ , and

P(\text{Type I error}) = P(X \le 2 \mid p = 0.3) = 0.0355.

(b) If $p = 0.1$ , then $X \sim B(20, 0.1)$ . A Type II error occurs if $X$ is not in the critical region, that is $X \ge 3$ .

P(X \ge 3) = 1 - \left(0.9^{20} + 20(0.1)(0.9)^{19} + 190(0.1)^2(0.9)^{18}\right) = 1 - (0.1216 + 0.2702 + 0.2852) = 1 - 0.6769 = 0.323

A Poisson test

The number of accidents per month at a junction has had the distribution $\text{Po}(4)$ . After new signs are installed, the council will test at the $5\%$ significance level whether the accident rate has fallen, using the number of accidents in one month.

(a) Find the critical region and the probability of a Type I error.

(b) Find the probability of a Type II error if the mean number of accidents per month is now $1.5$ .

Solution

(a) $H_0: \lambda = 4$ , $\quad H_1: \lambda < 4$ . Under $H_0$ , $X \sim \text{Po}(4)$ .

P(X = 0) = e^{-4} = 0.0183 \le 0.05, \qquad P(X \le 1) = 5e^{-4} = 0.0916 > 0.05.

The critical region is $X = 0$ , and $P(\text{Type I error}) = 0.0183$ .

(b) If $\lambda = 1.5$ , then $X \sim \text{Po}(1.5)$ . A Type II error occurs if $X \ge 1$ .

P(X \ge 1) = 1 - e^{-1.5} = 0.777

This is large: with only one month of data, even a big fall in the accident rate will usually go undetected. Using several months of data would reduce it.

A normal test

Bags of sugar have masses $N(\mu, 8^2)$ grams, and $\mu$ should be $1000$ . A random sample of $16$ bags is weighed and $H_0: \mu = 1000$ is tested against $H_1: \mu < 1000$ at the $5\%$ significance level.

(a) State the probability of a Type I error.

(b) Find the probability of a Type II error if the true mean is $995$ g.

Solution

(a) The test statistic is normal, so $P(\text{Type I error}) = 0.05$ .

(b) Under $H_0$ , $\bar{X} \sim N\left(1000, \dfrac{64}{16}\right) = N(1000, 4)$ . Reject $H_0$ if

\bar{x} < 1000 - 1.645 \times 2 = 996.71.

If $\mu = 995$ , then $\bar{X} \sim N(995, 4)$ . A Type II error occurs if $\bar{X} \ge 996.71$ .

P(\bar{X} \ge 996.71) = P\left(Z \ge \frac{996.71 - 995}{2}\right) = P(Z \ge 0.855) = 1 - 0.8037 = 0.196

Which error, in context

In the sugar test above, a sample of $16$ bags has mean mass $995.9$ g.

(a) Carry out the test.

(b) State which type of error could have been made, and describe it in context.

Solution

(a) $995.9 < 996.71$ , so $\bar{x}$ is in the critical region. Reject $H_0$ : there is evidence at the $5\%$ level that the mean mass of the bags is less than $1000$ g.

(b) $H_0$ was rejected, so the only possible error is a Type I error: concluding that the mean mass of the bags is less than $1000$ g when in fact it is $1000$ g.

(A Type II error, here, would be concluding that there is no evidence the bags are underweight when in fact their mean mass is less than $1000$ g. It could not have happened in this test, because $H_0$ was rejected.)

Exam-hard: a two-tailed test and the sample size needed

The marks in a national test are $N(\mu, 5^2)$ . A test of $H_0: \mu = 50$ against $H_1: \mu \ne 50$ at the $5\%$ significance level uses the mean of a random sample of $n$ marks.

(a) For $n = 25$ , find the probability of a Type II error if in fact $\mu = 53$ .

(b) Find the smallest value of $n$ for which the probability of a Type II error, when $\mu = 53$ , is less than $0.01$ . You may neglect the lower tail of the critical region.

Solution

(a) Under $H_0$ , $\bar{X} \sim N(50, 1)$ . The acceptance region is

50 - 1.96 < \bar{x} < 50 + 1.96, \quad\text{that is}\quad 48.04 < \bar{x} < 51.96.

If $\mu = 53$ , $\bar{X} \sim N(53, 1)$ , and

P(48.04 < \bar{X} < 51.96) = P(-4.96 < Z < -1.04) = (1 - 0.8508) - 0.0000 = 0.149.

For a two-tailed test, the acceptance region has two boundaries, and the Type II probability is the area between them under the true distribution. Here the lower boundary is so far from $53$ that it contributes nothing.

(b) With sample size $n$ , the standard deviation of $\bar{X}$ is $\dfrac{5}{\sqrt{n}}$ , and the upper boundary of the acceptance region is $50 + 1.96 \times \dfrac{5}{\sqrt{n}}$ .

If $\mu = 53$ , we need

P\left(\bar{X} < 50 + \frac{9.8}{\sqrt{n}}\right) < 0.01 \quad\Longleftrightarrow\quad \frac{50 + 9.8/\sqrt{n} - 53}{5/\sqrt{n}} < -2.326.

Simplify the left-hand side: $1.96 - \dfrac{3\sqrt{n}}{5} < -2.326$ , so

\frac{3\sqrt{n}}{5} > 4.286 \;\Rightarrow\; \sqrt{n} > 7.143 \;\Rightarrow\; n > 51.03.

The smallest sample size is $n = 52$ .

Common mistakes

Using $H_0$ to find a Type II probability. The critical region comes from $H_0$ , but the probability of a Type II error is calculated with the true parameter value.
Saying $P(\text{Type I error}) = 5\%$ for a binomial or Poisson test. It is the actual significance level, such as $0.0355$ .
Using the critical region instead of the acceptance region for a Type II error. Type II means $H_0$ is not rejected, so you need the probability of the acceptance region.
Mixing up the definitions. Type I: reject a true $H_0$ . Type II: fail to reject a false $H_0$ . A useful memory aid: Type I is the error you control directly with the significance level.
Saying both errors were possible. In a particular test only one is: Type I if $H_0$ was rejected, Type II if not.
Forgetting the second boundary in a two-tailed Type II calculation. Check whether the far boundary contributes; usually it adds almost nothing, but say so.
Describing errors without context. "Rejecting $H_0$ when it is true" earns less than "concluding the mean mass is less than $1000$ g when it is $1000$ g".

Exam tip

"Explain what is meant by a Type I error in this context" wants the definition applied to the question: what would be concluded, and what is actually true.
"State which type of error could have been made" needs the reason: "Type I, because $H_0$ was rejected".
For binomial and Poisson tests, the Type I probability needs the critical region. If an earlier part used the probability method, you must now find the critical region.
For a Type II probability, write the true distribution explicitly, such as "if $p = 0.1$ , $X \sim B(20, 0.1)$ ", and state the event, such as "Type II error: $X \ge 3$ ".
Normal tests: the critical value of $\bar{x}$ (such as $996.71$ ) is the key intermediate value. Keep at least 4 significant figures in it.
The syllabus asks for error probabilities in tests based on a normal distribution or on direct evaluation of binomial or Poisson probabilities. Set the work out in the same way whichever of these the question uses: critical region from $H_0$ , then the probability under the stated true value.

Summary

Type I error: rejecting $H_0$ when it is true. Type II error: not rejecting $H_0$ when it is false.
$P(\text{Type I}) = P(\text{critical region} \mid H_0)$ : equal to $\alpha$ for a normal test, the actual significance level for a discrete test.
$P(\text{Type II}) = P(\text{acceptance region} \mid \text{true value})$ ; a specific true value is needed.
Find the critical region using $H_0$ ; find the Type II probability using the true distribution.
Only one error is possible in a given test: Type I if $H_0$ was rejected, Type II if not.
Lowering $\alpha$ reduces Type I but increases Type II; a larger sample reduces Type II for the same $\alpha$ .
Describe errors in the context of the question.

Practice questions

Question

A test of $H_0: p = 0.4$ against $H_1: p > 0.4$ is carried out at the $5\%$ significance level, using $X \sim B(15, p)$ . Find the critical region and the probability of a Type I error.
For the test in question 1, find the probability of a Type II error if in fact $p = 0.7$ .
A test of $H_0: \lambda = 6$ against $H_1: \lambda > 6$ is carried out at the $5\%$ significance level, using a single observation of $X \sim \text{Po}(\lambda)$ . Find the probability of a Type II error if in fact $\lambda = 10$ .
$X \sim N(\mu, 5^2)$ . A test of $H_0: \mu = 50$ against $H_1: \mu > 50$ at the $5\%$ significance level uses the mean of a random sample of $25$ observations. Find the probability of a Type II error if in fact $\mu = 53$ .
A drug company tests whether a new drug cures a higher proportion of patients than the old one. The test result is to reject $H_0$ . State which type of error could have been made, and describe it in context.
(a) Explain why the probability of a Type II error cannot be found without a specific value of the parameter. (b) Explain why reducing the significance level of a test increases the probability of a Type II error.
A test of $H_0: p = 0.3$ against $H_1: p \ne 0.3$ uses $X \sim B(20, p)$ and has critical region $X \le 1$ or $X \ge 11$ . Find the probability of a Type I error, and the probability of a Type II error if in fact $p = 0.5$ .
Breakdowns of a machine occur at random at an average rate of $3$ per week. After an overhaul, a manager will test at the $5\%$ significance level whether the rate has decreased, using the total number of breakdowns in $3$ weeks. (a) Find the critical region and the probability of a Type I error. (b) Find the probability of a Type II error if the true rate after the overhaul is $1.5$ per week. (c) The manager's assistant suggests using just $1$ week instead. Find the critical region and the probability of a Type II error in this case, and comment.

Answers

Under $H_0$ , $X \sim B(15, 0.4)$ . $P(X \ge 10) = 0.0338 \le 0.05$ and $P(X \ge 9) = 0.0950 > 0.05$ . Critical region $X \ge 10$ ; $P(\text{Type I error}) = 0.0338$ .
If $p = 0.7$ , $X \sim B(15, 0.7)$ . Type II error: $X \le 9$ . $P(X \le 9) = 0.278$ .
Under $H_0$ , $X \sim \text{Po}(6)$ : $P(X \ge 11) = 0.0426 \le 0.05$ and $P(X \ge 10) = 0.0839 > 0.05$ , so the critical region is $X \ge 11$ . If $\lambda = 10$ , $X \sim \text{Po}(10)$ . Type II error: $X \le 10$ . $P(X \le 10) = 0.583$ .
Under $H_0$ , $\bar{X} \sim N(50, 1)$ . Reject if $\bar{x} > 50 + 1.645 = 51.645$ . If $\mu = 53$ , $\bar{X} \sim N(53, 1)$ . $P(\bar{X} < 51.645) = P(Z < -1.355) = 1 - 0.9123 = 0.0877$ .
$H_0$ was rejected, so a Type I error could have been made: concluding that the new drug cures a higher proportion of patients when in fact it cures the same proportion as the old one.
(a) A Type II error happens when $H_0$ is false, and its probability is the probability of the acceptance region under the true distribution. " $H_0$ is false" does not say what the true value is, and the probability depends on it (it is larger when the true value is close to the $H_0$ value), so a specific value is needed. (b) A smaller significance level makes the critical region smaller, so the acceptance region is larger. A larger acceptance region has a larger probability under the true distribution, so a Type II error becomes more likely.
$P(\text{Type I}) = P(X \le 1 \mid p = 0.3) + P(X \ge 11 \mid p = 0.3) = 0.0076 + 0.0171 = 0.0248$ . If $p = 0.5$ , $X \sim B(20, 0.5)$ . Type II error: $2 \le X \le 10$ . $P(2 \le X \le 10) = P(X \le 10) - P(X \le 1) = 0.5881 - 0.0000 = 0.588$ .
(a) Over $3$ weeks, under $H_0$ , $X \sim \text{Po}(9)$ . $P(X \le 3) = 0.0212 \le 0.05$ and $P(X \le 4) = 0.0550 > 0.05$ . Critical region $X \le 3$ ; $P(\text{Type I error}) = 0.0212$ . (b) True mean over $3$ weeks $= 4.5$ , so $X \sim \text{Po}(4.5)$ . Type II error: $X \ge 4$ . $P(X \ge 4) = 1 - e^{-4.5}\left(1 + 4.5 + \tfrac{4.5^2}{2} + \tfrac{4.5^3}{6}\right) = 1 - 0.3423 = 0.658$ . (c) For $1$ week, under $H_0$ , $X \sim \text{Po}(3)$ . $P(X = 0) = e^{-3} = 0.0498 \le 0.05$ and $P(X \le 1) = 4e^{-3} = 0.199 > 0.05$ , so the critical region is $X = 0$ . If the true rate is $1.5$ per week, $X \sim \text{Po}(1.5)$ , and $P(\text{Type II}) = P(X \ge 1) = 1 - e^{-1.5} = 0.777$ . The one-week test is much more likely to miss a genuine decrease ( $0.777$ against $0.658$ ), so the three-week test is better. Longer observation periods give the test more power to detect a change.

Two ways to be wrong

The probability of a Type I error

The probability of a Type II error

Practice questions

Console