Rough draft.This research track is under review with Dr. Atit's lab. Content and sequence may still change.
The Baby Mateo Case
Experimental Design domainBiomedical Innovations (BI)Lesson 17 of 20Your seat: Data Scientist
How Do We Combine Many Studies Into One Answer?
Discovery question
How do we combine many separate studies into one trustworthy answer, without cherry-picking?
💡 A uses PRISMA to find and judge all the studies on a question, and a meta-analysis pools only the studies that are similar enough; too much heterogeneity means do not combine.
The plan
Prerequisite check
Before this page, you should know
- Bias is a built-in error in how a study is run or measured that pushes results one direction; (a measurement bias) and survival bias (a selection bias) are two examples.
- Confounding happens when a hidden third variable is tangled with both the thing you changed and the thing you measured, so it can masquerade as the cause.
If not, review
Today's new idea is only
A uses PRISMA to find and judge all the studies on a question, and a meta-analysis pools only the studies that are similar enough; too much heterogeneity means do not combine.
Learn first
What you will learn
Goal: Explain how a pools studies using PRISMA, read a PRISMA flow diagram, and state when studies are too different to combine (heterogeneity).
Know by the end
- A is a structured, repeatable way to find all the studies on one question, judge their quality, and summarize them, reported under the 27-item checklist plus a flow diagram.
- Gold-standard reviews use a pre-registered protocol, a multi-database search, dual independent screening, a formal risk-of-bias check, and an assessment of .
- A meta-analysis is the statistics step that pools comparable studies into one summary estimate with a 95% CI and measures heterogeneity (often as I-squared); too-heterogeneous studies should not be pooled.
- The TOPS team noted an earlier of -timing trials could not statistically combine its four trials because they were too different (different ages, techniques, outcome definitions).
Learn first
Model: A PRISMA flow diagram, and a real moment when studies would not pool
Every trustworthy reports its search as a flow diagram required by . Here is a worked example with realistic teaching numbers for a -timing question (the numbers are constructed for teaching; the PRISMA structure is real). Records found by database search: 842. Duplicates removed: minus 310, leaving 532 records screened by title and abstract. Excluded as not about cleft timing: minus 489, leaving 43 full-text articles assessed. Excluded for wrong design or no usable data: minus 39, leaving 4 studies included in the review. How many of those 4 have poolable data for a meta-analysis is the open question.
The TOPS trial team noted that an earlier of timing trials could not statistically combine its four trials, because the trials were too different from each other to pool into one number. Different ages, different techniques, different outcome definitions: combining them would average apples and oranges.
Do the work
Explore (work the model before reading on)
- How many records did the search start with, and how many studies survived to be included?
- Name two reasons studies were thrown out in the flow diagram.
- How many trials did the earlier review find, and what could it not do with them?
- Why does writing down every excluded study make a review more trustworthy than just listing the studies an author liked?
- Suppose a review pooled four trials that used totally different speech tests into one average. Predict how that could produce a precise-looking number that is actually misleading.
The plan
Guided notes
1
Systematic review and PRISMA
Model start: A finds and appraises all the studies on one question in a repeatable way, so the answer does not depend on which papers an author happened to like.
- A is a structured, repeatable way to find ____ the studies on one question, judge their quality, and summarize them.
- The is , a 27-item checklist plus a flow ____.
- Gold-standard features include a pre-registered protocol, a multi-database search, dual independent screening, a formal ____-of-bias check, and an assessment of .
2
Meta-analysis and heterogeneity
- A meta-analysis pools the included studies into one summary estimate with a 95% ____ interval and measures heterogeneity, often reported as I-squared.
- If studies are too ____, you should not pool them, exactly the call the -timing reviewers made.
- The danger to remember: garbage in, garbage out. Pooling biased studies gives a precise-looking but ____ answer.
Explore
Reading the Research
What to read
Read the title and the abstract only, not the whole paper. Page et al. 2021, The PRISMA 2020 statement (BMJ)
Why this source matters
This is the published evidence behind today's idea: A uses PRISMA to find and judge all the studies on a question, and a meta-analysis pools only the studies that are similar enough; too much heterogeneity means do not combine.
Words to unlock first
systematic reviewmeta-analysisheterogeneityPRISMA 2020publication bias
Reading moves
- Skim the title and abstract first to get the gist.
- Circle the one sentence that states the main claim.
- Box the evidence the authors give for that claim.
- Mark one sentence that confuses you, and move on.
Stop point
You do not need the methods or statistics yet. If a sentence is about lab technique or math you have not learned, mark it and skip it.
Your output
Write one claim-evidence sentence: what this source claims, and the one piece of evidence that backs it up.
Where this fits
Tested on (Ohio WebXam)
Genetics of Disease · 072130
PLTW lesson
MI · Experimental Design domain · Systematic review and meta-analysis; PRISMA, heterogeneity, and publication bias
WebXam domain
Bio-Molecular Technology
Evidence to produce
Use the flow diagram. Of the 4 included studies, suppose 3 measured speech with the same validated test and 1 used a completely different home-made test. As Data Scientist: (1) which studies could you reasonably pool, and which one would you keep separate, and why? (2) fill the '?' in the diagram: how many studies go into the meta-analysis under your decision? (3) write one sentence telling the PI what the review can honestly conclude.
Lab / skill
Biomedical Innovations (BI) · AP Biology
Words
Vocabulary (the same words your classes use)
Explore
Vetted readings for this lesson
The plan
Track your progress today
Check these off as you work through the lesson, then submit. This tells Mr. Mendoza how you're doing so he can help the class. It does not replace turning in your producible.
Use the code Mr. Mendoza gave you, not your name. Saved on this device.
Check off as you finish
- Read the Model and answered the Explore questions.
- Filled in the guided notes in my own words.
- Defined the new vocabulary with an example.
- Built the producible: Use the flow diagram. Of the 4 included studies, suppose 3 measured speech with the same validated test and 1 used a completely different home-made test. As Data Scientist: (1) which studies could you reasonably pool, and which one would you keep separate, and why? (2) fill the '?' in the diagram: how many studies go into the meta-analysis under your decision? (3) write one sentence telling the PI what the review can honestly conclude.
- Wrote my Claim, Evidence, and Reasoning exit ticket.
Pick your period and code first.
Check yourself
Exit ticket (Claim, Evidence, Reasoning)
- Claim: Combining many studies can give a stronger answer than any single study, but only when it is done carefully.
- Evidence: requires ____ (name one safeguard), and a real -timing review could not pool its four trials because they were too ____.
- Reasoning: Explain why a careless meta-analysis of biased or mismatched studies can be worse than honest single studies.
How this is graded (rubric)
For: Use the flow diagram. Of the 4 included studies, suppose 3 measured speech with the same validated test and 1 used a completely different home-made test. As Data Scientist: (1) which studies could you reasonably pool, and which one would you keep separate, and why? (2) fill the '?' in the diagram: how many studies go into the meta-analysis under your decision? (3) write one sentence telling the PI what the review can honestly conclude.
| Criterion | Proficient | Developing | Beginning |
|---|---|---|---|
| Complete | Every required part of the artifact is present and filled in. | Most parts are present, but one is missing or left blank. | Several parts are missing. |
| Accurate | The science and data are correct and match the evidence. | Mostly correct, with a small factual slip. | Key science or data is wrong. |
| Scientific reasoning (CER) | States a claim, backs it with specific evidence, and explains the reasoning. | Has a claim and evidence, but the reasoning is thin or missing. | Gives an answer with no evidence or reasoning. |
| Professional communication | Clear, organized, and labeled the way a clinician or scientist would write it. | Readable but disorganized or missing labels. | Hard to follow. |
| Submitted | Turned in the right way (Schoology for routine work) and confirmed. | Turned in, but in the wrong place or unconfirmed. | Not turned in. |
How the model answer scores against this rubric
- CompleteProficient: Nothing is left blank: the model fills every part of "Use the flow diagram. Of the 4 included studies, suppose 3 measured speech with the same validated test and 1 used a completely different home-made test. As Data Scientist: (1) which studies could you reasonably pool, and which one would you keep separate, and why? (2) fill the '?' in the diagram: how many studies go into the meta-analysis under your decision? (3) write one sentence telling the PI what the review can honestly conclude.".
- AccurateProficient: Every number and claim matches the case evidence.
- Scientific reasoning (CER)Proficient: It names a claim, cites the specific evidence, and explains the reasoning, not just the answer.
- Professional communicationProficient: It is organized and labeled like a real chart note.
- SubmittedProficient: It would be turned in on Schoology and confirmed.
Explore
Where this leads: careers
Data Scientist Research Synthesis Methodologist Evidence-Based Medicine Researcher
What's next: Combining studies only works if the studies were done on real children fairly. But every study we pool was run on real babies and families. What rules protect children in research in the first place, and who decides what is allowed? We chase that next time.