How Does the World Check That a Study Is Trustworthy?
Once a study is finished, how does the rest of the world check whether it is actually trustworthy?
💡 Complete reporting (CONSORT, STROBE, PRISMA, ARRIVE) is what makes reproducibility and possible; an unreported step is, to the outside world, the same as a missing step.
Prerequisite check
- An IRB or ethics committee is an independent group that must approve a study's protocol before it begins and approve any later changes; TOPS needed approval in every one of its five countries.
- means the participant, or for a child a parent or guardian, is told the real risks, benefits, and alternatives and freely agrees in writing, with the right to withdraw; an infant cannot consent, so parents give permission.
What you will learn
Goal: Explain how reproducibility, reporting standards (CONSORT for trials, STROBE for observational studies), and let outsiders check a study, and use a reporting checklist to spot what a weak paper left out.
- Audits of animal papers found randomization reported in only about 30 to 40%, in roughly 20%, and a sample-size justification in under 10%, which is why reporting standards exist.
- Each study type has its own reporting checklist: CONSORT for randomized trials, STROBE for observational studies, PRISMA for systematic reviews, and ARRIVE for animal research.
- Reproducibility means a result holds up when other people, using your written methods, run the study again and get the same finding; a result no one can repeat is not yet knowledge.
- is independent experts reading and judging a paper before publication; it works only when the paper reports enough detail, which is exactly what the standards force.
Model: Why reporting guidelines exist, and a trial reported so anyone could check it
When reviewers audited published animal studies, they found something alarming: randomization was reported in only about 30 to 40 percent of papers, in roughly 20 percent, and a sample-size justification in under 10 percent. The studies may have done these things, but the papers did not say so, which means no outside reader could check. That gap is why the science community built reporting standards: shared checklists of what every paper must state. The major checklists, one per study type, are CONSORT for a , STROBE for an (cohort or case-control), PRISMA for a and meta-analysis, and ARRIVE for animal research.
The TOPS trial published, in plain view, exactly what a CONSORT-style report demands: how babies were randomized (a web-based minimization algorithm with a random element, stratified by surgeon and extent), who was blinded (speech assessors did not know which timing group a child was in), the pre-set and the behind it (292 per group for 80 percent power), the one chosen in advance, and the result with its uncertainty ( 0.59, 95 percent CI 0.36 to 0.99). Because all of that is on the page, any outside scientist can judge the study without taking the authors' word for it.
Explore (work the model before reading on)
- What fraction of animal papers reported , and what fraction justified their ?
- Which checklist is used for a randomized trial, and which for a ?
- List two specific things TOPS reported that let an outsider check the study.
- A study could have been done well but reported badly. Explain why a reader still cannot trust a study whose paper leaves out how it randomized or blinded.
- Two teams run the same experiment and get opposite results. Predict what reporting and repeating let the community do that a single paper alone never could.
Guided notes
Reproducibility and reporting standards
- Reproducibility means a result holds up when other people, using your written methods, run the study again and get the ____ finding.
- A is a shared ____ of everything a paper must state so an outsider can fully understand and re-run it.
- The checklists by study type are CONSORT for randomized trials, STROBE for ____ studies, PRISMA for systematic reviews, and ARRIVE for animal research.
Peer review
- is the step where independent ____, who did not do the study, read the paper before a journal publishes it and can demand changes or reject it.
- is a first ____ that catches obvious flaws, not a guarantee that a study is correct.
- It works only when the paper reports enough ____ for reviewers to judge it, which is exactly what the reporting standards force.
Reading the Research
- Skim the title and abstract first to get the gist.
- Circle the one sentence that states the main claim.
- Box the evidence the authors give for that claim.
- Mark one sentence that confuses you, and move on.
Vocabulary (the same words your classes use)
Track your progress today
Check these off as you work through the lesson, then submit. This tells Mr. Mendoza how you're doing so he can help the class. It does not replace turning in your producible.
Use the code Mr. Mendoza gave you, not your name. Saved on this device.
- Read the Model and answered the Explore questions.
- Filled in the guided notes in my own words.
- Defined the new vocabulary with an example.
- Built the producible: A draft cleft paper crosses your desk: 'We operated on some babies with clefts at different ages and the early ones spoke better. The difference looked big.' As Science Writer: (1) name three things a CONSORT-style standard requires that this paper left out; (2) explain why a peer reviewer could not trust this paper as written; (3) rewrite the sentence so it reports one missing item properly (sample size, randomization, blinding, or the result with a confidence interval), using TOPS as your model.
- Wrote my Claim, Evidence, and Reasoning exit ticket.
Exit ticket (Claim, Evidence, Reasoning)
- Claim: A study becomes trustworthy not when its authors say so, but when outsiders can check, repeat, and review it.
- Evidence: Audits found randomization and were reported in only a minority of animal papers, which is why reporting standards exist.
- Reasoning: Explain why complete reporting is what makes both reproducibility and possible.
| Criterion | Proficient | Developing | Beginning |
|---|---|---|---|
| Complete | Every required part of the artifact is present and filled in. | Most parts are present, but one is missing or left blank. | Several parts are missing. |
| Accurate | The science and data are correct and match the evidence. | Mostly correct, with a small factual slip. | Key science or data is wrong. |
| Scientific reasoning (CER) | States a claim, backs it with specific evidence, and explains the reasoning. | Has a claim and evidence, but the reasoning is thin or missing. | Gives an answer with no evidence or reasoning. |
| Professional communication | Clear, organized, and labeled the way a clinician or scientist would write it. | Readable but disorganized or missing labels. | Hard to follow. |
| Submitted | Turned in the right way (Schoology for routine work) and confirmed. | Turned in, but in the wrong place or unconfirmed. | Not turned in. |
- CompleteProficient: Nothing is left blank: the model fills every part of "A draft cleft paper crosses your desk: 'We operated on some babies with clefts at different ages and the early ones spoke better. The difference looked big.' As Science Writer: (1) name three things a CONSORT-style standard requires that this paper left out; (2) explain why a peer reviewer could not trust this paper as written; (3) rewrite the sentence so it reports one missing item properly (sample size, randomization, blinding, or the result with a confidence interval), using TOPS as your model.".
- AccurateProficient: Every number and claim matches the case evidence.
- Scientific reasoning (CER)Proficient: It names a claim, cites the specific evidence, and explains the reasoning, not just the answer.
- Professional communicationProficient: It is organized and labeled like a real chart note.
- SubmittedProficient: It would be turned in on Schoology and confirmed.
Where this leads: careers
What's next: We answered today's question: reporting standards, reproducibility, and let the world check a study instead of trusting it on faith. Across this whole domain you have learned to ask a , build a hypothesis, pick a design, control bias, run the stats, clear ethics, and report it all. So here is the last test: what new question about Mateo's would YOU investigate, and how would you design a study to answer it? We chase that next time, and this time the study is yours.