Experimental design and data analysis

What the exam asks

The commonest items are: which of these comparisons tests the hypothesis, what does the control tube tell you, what further measurement would distinguish two explanations the stem has given, and does the conclusion offered follow from the data. The trap that catches most candidates is accepting a conclusion the data is consistent with instead of the one the data forces. Three distractors will usually be perfectly compatible with the results and still unsupported by them, so 'could this be true?' is the wrong test and will pick a wrong answer; the test is 'does this experiment rule out the alternatives?'. The second trap is the single-point design: when a stem measures one condition at one value and asks which mechanism is at work, the honest answer is almost always that this experiment cannot say, and the right option is the one naming the extra reading that would.

This is not a topic with content of its own. It is the method every other topic in the section is examined through. A stem gives you an experiment and a result, and the question is about the experiment: which comparison was made, what it can support, and what it cannot. You will meet it in a biology stem, a chemistry stem and a physics stem, and it behaves identically in all three.

One move does most of the work. Take the two conditions being compared and list every way they differ. If they differ in one thing, the comparison isolates that thing. If they differ in two, the comparison isolates nothing, no matter how large the difference in the result or how many times it was repeated. Most wrong answers in this section are conclusions the data is merely consistent with, offered in place of the one the data forces.

The second move is reading the result honestly. A result can be real and trivial, or large and unreliable, and these are separate questions answered by separate parts of the data. Sample size and statistics speak to whether a difference is likely to be chance. Nothing about them speaks to whether it matters, and nothing about them repairs a design that was measuring the wrong thing.

What you carry in is a set of questions rather than facts: what does the control supply, what would each hypothesis predict here, is there a reading that separates them, and does the conclusion on offer go further than the measurement does.

What to hold

• A comparison isolates a variable only if the conditions being compared differ in that variable and nothing else.
• A control group supplies the counterfactual: what the result would have been without the treatment. Without it a change has nothing to be a change from.
• A negative control shows the signal does not appear on its own; a positive control shows the assay would have detected the effect if it were there, which is the only thing that makes a negative result meaningful.
• A confounder is a variable that tracks the one you changed, so its effect and your variable's effect arrive at the result together and cannot be separated after the fact.
• Randomisation matters because it spreads confounders you have not thought of, which is the only defence against the ones you cannot name.
• A design that takes one reading can show that something changed, but not what changed, because competing mechanisms differ in the shape of a curve and a single point has no shape.
• Initial rates are measured because they are taken before anything except the intended variable has moved, which is a general design principle and not a fact about enzymes.
• Statistical significance says a difference is unlikely to be a fluke of sampling. It says nothing whatever about the size of the difference.
• Increasing sample size shrinks random error and narrows the uncertainty on an estimate. It does nothing to bias, so a biased design with a huge sample is precisely wrong rather than approximately right.
• No significant difference means the study failed to detect one, which is not the same as showing there is none.
• A correlation is consistent with the causal claim, with the reverse causal claim, and with a third variable driving both. The data alone does not choose between them.
• A relationship is evidence only across the range that was measured; extending it past the last data point is an assumption, not a finding.
• A measurement both hypotheses predict cannot discriminate between them, however precisely it is made.