Is the Difference Real? Significance & p-values
Decide whether a difference between groups is a real effect or just the luck of the draw, using the p-value and the right test.
Almost every science claim comes down to one question: is this difference real, or could plain chance have produced it? Two groups will almost never come out exactly equal, even when nothing is going on, so we need a fair rule for deciding when a gap is big enough to trust. The p-value is that rule: it is the probability of seeing a difference this big (or bigger) if only chance were acting. By a long-standing convention, a p-value below 0.05 is called statistically significant, meaning chance alone is an unlikely explanation. Epidemiologists use this to decide whether a new outbreak rate is really higher than normal. Biostatisticians use it to judge whether a drug beat the placebo in a clinical trial. Lab scientists use the t-test to compare two group means and the chi-square test to check whether observed counts (like a genetics ratio) match what was expected. Doctors reading studies rely on it to separate treatments that truly help from results that were just noise. One caution runs through all of it: significant does not mean large or important, and a p-value is never absolute proof.
- Common Core · HSS-IC.A.1Understand that statistics lets you make inferences about a population from a random sample, and that conclusions carry uncertainty.
- Common Core · HSS-IC.B.5Compare two treatments or groups using data from an experiment, and decide whether the observed difference is larger than chance would reasonably produce.
- Ohio · Ohio HS S.IC.1Recognize that a sample gives evidence about a larger population and that any conclusion drawn from data includes a chance of being wrong.
- NGSS · SEP-4Analyzing and Interpreting Data: use statistical reasoning to distinguish a real signal in data from random variation before drawing a conclusion.
- AP · AP Bio SP 5 (Statistical Tests)Perform and interpret a chi-square test, comparing a calculated value to a critical value to decide whether to reject the null hypothesis.
- Find the mean (average) of a data set: A t-test compares two group means, so students must be able to compute an average before they can compare two.
- Read a probability as a number from 0 to 1: A p-value is a probability; students must know that 0.05 means about a 5 in 100 chance of seeing a gap this big when nothing real is going on, which is a statement about chance, not about how much of the data changed.
- Tell observed counts from expected counts: The chi-square test compares what you counted to what a ratio predicts, so students must keep observed and expected apart.
Prerequisites are inferred: pending teacher review.
Re-learn the skill with worked practice and clear examples.
Use the significance rule: compare your p-value to the 0.05 cutoff. If p is below 0.05, chance is an unlikely explanation, so you call the difference statistically significant; if p is 0.05 or above, you do not have enough evidence to rule out chance. Then match the test to the data: a t-test for comparing two means, a chi-square test for comparing observed counts to expected counts.
A study reports p = 0.03 for the difference between two groups, using the usual 0.05 cutoff. What is the correct conclusion?
Reviewed- A.The result is not significant; keep the null hypothesis.
- B.The result is statistically significant; chance is an unlikely explanation.
- C.The difference is proven true beyond any doubt.
- D.The difference must be large and important.
Show the worked solution ▾
Answer: B. The result is statistically significant; chance is an unlikely explanation.
- Step 1: Compare to the cutoff: 0.03 is less than 0.05, so it falls in the significant zone shown left of the cutoff.
- Step 2: State the conclusion carefully: Below 0.05 means chance is an unlikely explanation, so we call the difference statistically significant, without claiming proof or importance.
Why it's right: Because 0.03 is below the 0.05 cutoff, the result is statistically significant, meaning chance alone is an unlikely explanation.
- A: 0.03 is below 0.05, so the result is significant, not non-significant.
- C: A small p-value never gives absolute proof; it only makes chance unlikely.
- D: Significant means chance is unlikely, not that the effect is large or important.
Aligned to Common Core HSS-IC.B.5: apply the significance rule · reading level ~grade 9
A scientist wants to know whether fertilized plants have a different mean height than unfertilized plants. Which test fits this comparison?
Reviewed- A.A chi-square test, because heights are counts.
- B.A t-test, because it compares two group means.
- C.No test is needed; just report the taller group.
- D.A t-test, but only if the p-value is already below 0.05.
Show the worked solution ▾
Answer: B. A t-test, because it compares two group means.
- Step 1: Name the kind of data: Height is a measured amount, and the scientist is comparing the average of one group to the average of another.
- Step 2: Match the test: Comparing two group means is exactly what a t-test does.
Why it's right: A t-test is the test for comparing the means of two groups, which is what mean height for two plant groups requires.
- A: Heights are measured amounts, not category counts, so chi-square does not fit.
- C: Reporting the taller group ignores whether the gap could be chance, which is the whole point.
- D: You run the test to find the p-value; you do not need the p-value before choosing the test.
Aligned to NGSS SEP-4: choose an appropriate statistical test · reading level ~grade 9
A geneticist crosses pea plants and predicts a 3 to 1 ratio of tall to short. She counts the offspring and wants to know if the counts fit that ratio. Which test fits, and what does it compare?
Reviewed- A.A t-test, comparing the two group means.
- B.A chi-square test, comparing observed counts to expected counts.
- C.A p-value, comparing tall plants to short plants directly.
- D.A chi-square test, comparing two different means.
Show the worked solution ▾
Answer: B. A chi-square test, comparing observed counts to expected counts.
- Step 1: Name the data: The data are counts in categories (tall, short) with an expected ratio of 3 to 1.
- Step 2: Match the test: A chi-square test compares observed counts to the counts you expected from the ratio, so it is the right choice.
Why it's right: A chi-square test compares observed counts to expected counts, which is exactly what checking a 3:1 genetics ratio needs.
- A: A t-test compares means of measured data, not counts in categories.
- C: A p-value is a result of a test, not a test you run, and it is not a comparison of the plants directly.
- D: Chi-square compares observed and expected counts, not two means.
Aligned to AP Bio SP 5: chi-square for a genetics ratio · reading level ~grade 9
- A lab group compares the mean reaction time of two conditions and runs a t-test to see whether the gap beats chance.
- A genetics class counts flower colors and uses a chi-square test to check a predicted 3:1 ratio.
- A student reads 'p = 0.03' in a study and correctly states that chance is an unlikely explanation, not that the effect is huge.
Fill these in as you work through the lesson.
- p-value (chance of a gap this big if only chance acts):
- Statistically significant (p below the 0.05 cutoff):
- t-test (compares two group means):
- chi-square test (compares observed counts to expected counts):
A result is called statistically significant when its p-value is 0.05, which means is an unlikely explanation; but significant does not mean the effect is or important, and it is never absolute .
- In your own words, what does a p-value of 0.02 tell you?
- You compare the mean height of two plant groups. Which test do you use, and why?
- Why can a very large sample make a tiny, unimportant difference come out statistically significant?
A cross of 120 plants expects 90 tall and 30 short. You observe 84 tall and 36 short. Tall term: (84 minus 90) squared divided by 90 = ____. Short term: (36 minus 30) squared divided by 30 = ____. Chi-square = ____ + ____ = ____.
The vocabulary of this topic, shown in the way you will meet it.