The Nature of Hypothesis Testing

A hypothesis test is a formal way of deciding whether data give real evidence against a claim, or whether what you observed could easily have happened by chance. A manufacturer claims that only $5\%$ of its components are faulty; a sample has $9$ faulty out of $80$. Is the claim false, or was this an unlucky sample? Every S2 test answers a question like this with the same logic and the same written structure, and every Paper 6 has at least one full test, usually worth $5$ or $6$ marks. This note sets out the language and the logic; the notes that follow apply them to binomial, Poisson and normal tests.

The logic of a test

Think of a courtroom. The defendant is presumed innocent, and is convicted only if the evidence would be very unlikely if they were innocent. A hypothesis test works the same way.

1. Start by assuming the claim is true. This is the null hypothesis.
2. Ask how likely a result at least as extreme as the one observed would be, if the claim were true.
3. If that probability is very small (below a threshold fixed in advance), the data are hard to explain under the claim, so reject it. Otherwise, the data are consistent with the claim and you do not reject it.

Here is the whole idea in one small example. A coin is claimed to be fair, but you suspect it favours heads. You toss it $10$ times and get $9$ heads. If the coin really is fair, the number of heads $X$ satisfies $X \sim B(10, 0.5)$, and

$$P(X \ge 9) = \frac{\binom{10}{9} + \binom{10}{10}}{2^{10}} = \frac{11}{1024} = 0.0107.$$

Nine or more heads would happen only about $1\%$ of the time with a fair coin. That is strong evidence that the coin is biased towards heads. It is not proof: fair coins do sometimes give $9$ heads in $10$ tosses. A test controls how often this kind of wrong conclusion happens, but cannot rule it out.

The vocabulary

The syllabus lists the terms you must understand and use. Learn them precisely; definitions are sometimes asked for directly.

Writing the hypotheses

Hypotheses are always about the population parameter, never about the sample. "$H_0: p = 0.1$" is about the proportion of the whole population; the sample result, such as $2$ out of $50$, is the evidence.

The question's wording tells you which alternative to use.

Wording in the question	Alternative hypothesis	Tails
"has increased", "is more than", "is biased towards", "underestimates"	$H_1: p > p_0$	One (upper)
"has decreased", "is less than", "has improved" (when lower is better)	$H_1: p < p_0$	One (lower)
"has changed", "is different", "is biased" (no direction)	$H_1: p \ne p_0$	Two

The direction must be decided by the question, before looking at the data. Choosing a one-tailed test because the sample happens to be high is not allowed.

Two ways to reach a decision

Once $H_0$, $H_1$ and the distribution of the test statistic under $H_0$ are set out, there are two equivalent ways to decide.

Compare a probability with the significance level. Calculate the probability, assuming $H_0$, of a result at least as extreme as the one observed, in the direction of $H_1$. If it is less than the significance level, reject $H_0$. (This probability is often called the $p$-value, though Paper 6 does not require the term.) For a two-tailed test, compare the probability in the observed tail with half the significance level.

Find the critical region. Work out which values of the test statistic would lead to rejection, then see whether the observed value is among them. This is essential when a question asks for the critical region, or for the probability of a Type I or Type II error.

For a test based on a normal distribution, the critical region is in one or both tails of the curve. The first graph shows a two-tailed test at the $5\%$ level, with $2.5\%$ in each tail beyond $z = \pm 1.96$; the second shows a one-tailed (upper) test at the $5\%$ level, with the whole $5\%$ beyond $z = 1.645$.

Discrete test statistics

For a binomial or Poisson test statistic, probabilities jump in steps, so a critical region usually cannot have probability exactly $5\%$. The critical region is chosen to have probability as close as possible to, but not more than, the significance level. Its actual probability is called the actual significance level of the test. You will see this in detail in tests for a binomial proportion.

Writing the conclusion

The conclusion has two parts: the decision about $H_0$, and what it means in the context of the question. Both are needed for the final mark.

• If the result is in the critical region: "Reject $H_0$. There is evidence at the $5\%$ level that the coin is biased towards heads."
• If not: "Do not reject $H_0$. There is insufficient evidence at the $5\%$ level that the coin is biased towards heads." Cambridge mark schemes also accept "accept $H_0$", provided the contextual statement is not definite.

Never claim certainty. A test does not prove that $H_0$ is false, and failing to reject $H_0$ does not show that it is true; it only means the data are consistent with it. Avoid words like "proves", "shows that" and "the coin is fair".

Which test, which distribution

Situation	Test statistic	Distribution under $H_0$	Note
Proportion, single observation of a count of successes	Number of successes $X$	$B(n, p_0)$	Binomial proportion
Rate of random events	Number of events $X$	$\text{Po}(\lambda_0)$	Poisson mean
Large $n$ or large $\lambda$	$X$, with continuity correction	Normal approximation	Normal approximations
Mean of a population	Sample mean $\bar{X}$	$N\left(\mu_0, \dfrac{\sigma^2}{n}\right)$	Population mean

Practice questions

Answers

1. (a) $p$ = proportion of the company's seeds that germinate. $H_0: p = 0.85$, $H_1: p < 0.85$. (b) $\lambda$ = mean number of accidents per month. $H_0: \lambda = 2.5$, $H_1: \lambda \ne 2.5$. (c) $\mu$ = population mean mass of packets in grams. $H_0: \mu = 250$, $H_1: \mu > 250$.

2. It is the probability of rejecting the null hypothesis when it is in fact true (at most $5\%$ for a discrete test statistic). Equivalently, $H_0$ is rejected only if a result as extreme as the one observed would happen less than $5\%$ of the time when $H_0$ is true.

3. $p$ = probability of a head. $H_0: p = 0.5$, $H_1: p > 0.5$. Under $H_0$, $X \sim B(12, 0.5)$. $P(X \ge 10) = \dfrac{\binom{12}{10} + \binom{12}{11} + \binom{12}{12}}{2^{12}} = \dfrac{66 + 12 + 1}{4096} = 0.0193$. $0.0193 < 0.05$: reject $H_0$. There is evidence at the $5\%$ level that the coin is biased towards heads.

4. $H_0: p = 0.5$, $H_1: p \ne 0.5$. Under $H_0$, $X \sim B(12, 0.5)$. $P(X \ge 9) = \dfrac{220 + 66 + 12 + 1}{4096} = \dfrac{299}{4096} = 0.0730$. Two-tailed at $10\%$, so compare with $0.05$: $0.0730 > 0.05$. Do not reject $H_0$. There is insufficient evidence at the $10\%$ level that the coin is biased.

5. A test never proves anything; the conclusion should say there is evidence that the proportion has increased. Also, the test says nothing about the size of the new proportion: $0.4$ is a sample value, and the population proportion is not known to equal it.

6. Under $H_0$, $\bar{X} \sim N\left(50, \dfrac{36}{16}\right)$, standard deviation $1.5$. $z = \dfrac{47.5 - 50}{1.5} = -1.667$. (a) Critical value $-2.326$. Since $-1.667 > -2.326$, do not reject $H_0$: insufficient evidence at the $1\%$ level that the mean mass is less than $50$ g. (b) Critical value $-1.645$. Since $-1.667 < -1.645$, reject $H_0$: evidence at the $5\%$ level that the mean mass is less than $50$ g.

7. In a two-tailed test at $5\%$, each tail has $0.025$. Since $0.032 > 0.025$, do not reject $H_0$: there is insufficient evidence that the parameter has changed.

8. (a) $\lambda$ = mean number of faults per $500$ m. $H_0: \lambda = 4$, $H_1: \lambda < 4$. Under $H_0$, $X \sim \text{Po}(4)$. $P(X \le 1) = e^{-4}(1 + 4) = 0.0916 > 0.05$. Do not reject $H_0$: insufficient evidence at the $5\%$ level that the fault rate has decreased. (b) $P(X = 0) = e^{-4} = 0.0183 < 0.05$ and $P(X \le 1) = 0.0916 > 0.05$, so the critical region is $X = 0$.