Systematic Reviews and Meta-Analysis: Reading a Forest Plot and Choosing the Right Study Design

A Shortcut Called Meta-Analysis

Dr. Bushra Fatima arrived with a printout and the look of someone who had found a door nobody else had noticed.

“Sir, I have solved my publication problem. A meta-analysis. No patients, no proforma, no ERC delays, no follow-up. I download twenty papers, put the numbers in software, and it sits at the top of the evidence pyramid. One month, and I have the strongest paper in this room.”

Dr. Sumaira Talib looked up from her laptop. “Bushra, our joint review on infection prevention is still a folder of forty PDFs in the Zotero library. We have not even decided what question it answers.”

“Then today both of you will learn what a systematic review really is,” I said, “and why the hardest part of a meta-analysis is not the software. But first, Bushra, I want to show you a picture. It is about your own dissertation topic.”

The Logo Built From Bushra’s Topic

I put the Cochrane logo on the screen: a circle, and inside it a few horizontal lines and a small diamond.

“In 1972, Liggins and Howie in New Zealand published a controlled trial showing that a steroid given to mothers before preterm delivery reduced respiratory distress syndrome in their babies [1]. Over the next years, more trials followed. Some showed benefit, some were too small to show anything, and obstetricians kept arguing. Most did not give the steroid.”

“In 1990, Crowley, Chalmers and Keirse did something simple. They collected every controlled trial they could find, 12 trials with over 3,000 participants, assessed them in a planned way, and combined the results [2]. Together, the trials showed clearly that antenatal corticosteroids reduce respiratory distress syndrome and neonatal death. The answer had been sitting in the literature for years. Nobody had added it up.”

“Each line in that logo is one trial. The diamond is the combined result. That review, updated many times since, became the symbol of the organisation that produces systematic reviews. The latest Cochrane update includes 27 trials, and antenatal corticosteroids reduce neonatal death with a risk ratio of 0.78, 95% confidence interval 0.70 to 0.87, high-certainty evidence [3].”

Bushra was quiet for a moment. “Then why am I doing my study at all? The question is already answered.”

“Whether steroids work is answered. That is exactly why your synopsis rationale cannot say it is unknown. Read the same review. It says questions remain about very early and late preterm births, multiple pregnancies and mothers with hypertension [3]. Your cohort studies what happens to preterm babies in your hospital, with your timing and your patients. Your rationale must name what is still uncertain locally. A systematic review is the best place to find that out before you write a single line of a synopsis.”

Narrative Review or Systematic Review?

Dr. Hammad Ali raised his hand. “Sir, my consultant wrote a review article last year. Twelve pages, eighty references. Is that a systematic review?”

“Probably a narrative review. The difference is not length. It is method.”

	Narrative review	Systematic review
Question	Broad topic (for example, “management of gallstones”)	One focused question, usually in PICO form
Protocol	None	Written and registered before the search starts
Search	Papers the author knows or chooses	Several databases with a reported, repeatable search strategy
Selection	Not stated	Pre-set eligibility criteria, applied by two reviewers independently
Quality of studies	Rarely judged	Risk of bias assessed for every included study
Synthesis	The author’s summary and opinion	Structured, with or without a meta-analysis
Can another team repeat it?	No	Yes, that is the whole point

“And a meta-analysis is the statistical part,” I added. “It combines the numbers from several studies into one estimate. A systematic review may or may not contain a meta-analysis. A meta-analysis without a systematic review behind it is just an average of whatever papers you happened to download.”

The Steps, From Protocol to Flow Diagram

“So where do we start?” asked Sumaira.

“With the same skills you already have, used more strictly,” I said, and wrote the steps on the board.

Question: a focused PICO question, with the outcomes and study designs you will include decided in advance.
Check first: search PROSPERO and the Cochrane Library for an existing or ongoing review on the same question.
Protocol and registration: write the protocol and register it on PROSPERO, the free international register of systematic review protocols, before screening begins. Registration shows editors that you did not change the question after seeing the results.
Search: at least two databases (for example PubMed and the Cochrane Central Register of Controlled Trials), plus trial registers and reference lists. Save the full search string and the date for every database.
Screening: two reviewers screen titles and abstracts, then full texts, independently. Disagreements go to a third person. Free tools such as Rayyan make this easier.
Data extraction: a piloted form, filled independently by two reviewers.
Risk of bias: RoB 2 for randomised trials [4] and ROBINS-I for non-randomised studies, which you met in the quasi-experimental session.
Synthesis: meta-analysis if the studies are similar enough; a structured narrative synthesis if they are not.
Certainty of evidence: rate each main outcome with GRADE as high, moderate, low or very low.
Reporting: follow the PRISMA 2020 statement, a 27-item checklist with a flow diagram showing how many records were found, screened, excluded (with reasons) and included [5].

“The Cochrane Handbook explains every one of these steps in detail, and it is free to read online [6],” I said. “Bushra, count the steps that need two people working independently.”

She counted. “Screening, extraction, risk of bias. Three.” She looked at Sumaira. “So I cannot do it alone in a month.”

“No good systematic review is done alone. And a careful one usually takes several months, not one.”

How to Read a Forest Plot

Dr. Zunaira Malik had been studying the logo. “Sir, I see forest plots in every meta-analysis, but I only look at the diamond. What do the other parts mean?”

I drew one on the board, using the neonatal death outcome from the corticosteroid review [3].

Part of the plot	What it tells you
Each row	One study, with its numbers of events and participants in each group
The square	That study’s result (here, a risk ratio). A bigger square means the study carries more weight, usually because it is larger
The horizontal line	That study’s 95% confidence interval. Small studies have long lines
The vertical line	The line of no effect: 1 for a risk ratio or odds ratio, 0 for a mean difference
Left or right of the line	Read the axis labels. Here, left of 1 favours steroids, right favours the control group
The diamond	The pooled result. Its centre is the combined estimate (0.78) and its width is the confidence interval (0.70 to 0.87). If the diamond does not touch the line of no effect, the pooled result is statistically significant
The heterogeneity line at the bottom	How much the studies disagree with each other, usually reported as I²

“Notice something,” I said. “In the logo, several of the individual trial lines cross the line of no effect. On their own, those trials looked inconclusive. Combined, the answer is clear. That is the power of a meta-analysis: small studies that are each too weak to answer a question can answer it together.”

“And I²?” asked Zunaira.

“I² is the percentage of the variation between studies that is due to real differences rather than chance. Higgins and colleagues suggested 25%, 50% and 75% as rough markers of low, moderate and high heterogeneity [7]. High heterogeneity is not a calculation error. It is a question: why do these studies disagree? Different patients, different doses, different definitions of the outcome? When the studies are similar, a fixed-effect model assumes they all estimate one true effect. When they differ, a random-effects model allows the true effect to vary between studies. Most clinical meta-analyses use random effects.”

The Missing Studies

Dr. Junaid Rashid had been listening from his corner. “Sir, in fifteen years I have seen many small studies that found nothing. None of them was ever published. Doesn’t a meta-analysis only see the studies that got published?”

“That is publication bias, and you have described it exactly. Small studies with positive results get published; small studies with negative results stay in drawers. A meta-analysis of only the published studies then looks better than the truth. The funnel plot is one way to look for it: each study’s effect is plotted against its size. Large studies sit near the top, close to the true effect; small studies scatter more widely at the bottom. Without bias, the picture looks like an upside-down funnel. If one bottom corner is empty, small negative studies may be missing [8].”

“But be careful. The Cochrane Handbook advises using tests for funnel plot asymmetry only when a meta-analysis has at least 10 studies [6]. With five studies, a funnel plot is a guess. This is also why we search trial registers: a registered trial that never published its results is a missing study you can actually find.”

Garbage In, Garbage Out

Hammad had an idea. “Sir, I found three Pakistani studies on shoulder-tip pain after laparoscopic cholecystectomy. One measured pain at 6 hours, one at 24 hours, one just said ‘present or absent’. Can I combine them?”

“Statistically, the software will combine anything you give it. Clinically, you should not. Different outcome definitions at different times are not the same outcome. A meta-analysis of three poorly matched studies gives a precise-looking diamond around a meaningless number. In that situation, describe the studies in a structured table and explain why pooling was not appropriate. That is an honest result.”

“There is a second problem. If the included trials are biased, the meta-analysis simply adds up the bias. When the corticosteroid review was updated in 2020, the authors removed six trials that had been included before, because they did not meet pre-set trustworthiness criteria [3]. That is the standard a good review sets for itself.”

“And that is why GRADE matters. A pooled result is rated down for risk of bias, inconsistency between studies, indirectness, imprecision and publication bias [9]. ‘High certainty’ and ‘very low certainty’ can sit under diamonds of exactly the same size.”

I turned back to Bushra. “Now about your one-month plan. Ioannidis examined the growth of these papers and found that systematic reviews and meta-analyses had increased many times faster than other publications. By his estimate, more systematic reviews of trials are now published each year than new randomised trials, and many topics have several overlapping meta-analyses. In one example, 185 meta-analyses of antidepressants for depression were published between 2007 and 2014 [10]. Editors have seen hundreds of quick meta-analyses. They recognise one immediately.”

“So how do I know whether a published review is a good one?” asked Sumaira.

“Appraise it with AMSTAR 2 [11]. It asks, among other things, whether a protocol was registered in advance, whether the search was comprehensive, whether two people selected and extracted the studies, whether risk of bias was assessed and taken into account, and whether publication bias was considered. Use it on every review you plan to cite in your synopsis.”

Which Design? A One-Page Guide

Dr. Hassan Raza had been waiting for this moment all session. “Sir, we have now done so many designs. Just give me one page. Which design for which question?”

For once, I agreed with him. “Three questions decide almost every design.”

Am I describing or comparing? Describing one group needs no comparison group. Testing whether something causes or changes something else does.
If comparing, do I assign the exposure? If you decide who gets the treatment, it is experimental: randomised (RCT) or not randomised (quasi-experimental). If nature, habit or the patient decides, it is observational.
If observational, what do I select patients on? On exposure, then follow them for the outcome: cohort. On outcome, then look back at exposure: case-control. Both measured at one point in time: cross-sectional.

Then we filled in the table together, one row for each project in the room.

If your question is…	Design	Example from this room
How common is it, here and now?	Cross-sectional	Junaid: frequency of medication non-adherence in hypertensive patients in the medicine OPD
Is this exposure linked to a rare disease?	Case-control	Zunaira: areca nut and gutka chewing in patients with and without oral submucous fibrosis
What happens over time to people who were exposed?	Cohort	Bushra: neonatal respiratory outcomes in preterm babies whose mothers did or did not receive antenatal corticosteroids
Does treatment A work better than treatment B?	Randomised controlled trial	Hammad: low- versus standard-pressure pneumoperitoneum and shoulder-tip pain
Did a programme work, when randomisation is impossible?	Quasi-experimental	Junaid: the adherence counselling desk, Unit I versus Unit II
How good is a test against the gold standard?	Diagnostic accuracy	Sumaira: ultrasound for acute appendicitis against histopathology
What happens after a treatment, in one group?	Descriptive case series (single-group cohort)	Hassan: functional outcome after ORIF of distal radius fractures
Is this rare event worth telling others about?	Case report	Hammad: left-sided gallbladder at laparoscopic cholecystectomy
What does all the existing evidence say?	Systematic review, with or without meta-analysis	Sumaira and Bushra: their joint infection prevention review

“One rule above all,” I said. “The question chooses the design. Not the time you have, not the design your senior used, and not the design that seems easiest to publish. Bushra chose a meta-analysis because it looked fast. That is the wrong reason. Choose it when the question is ‘what does all the evidence say’, and when you have checked that nobody has answered it well already.”

Forty PDFs and One Question

Before the session ended, Sumaira and Bushra sat together over the Zotero library and wrote one PICO question: in women undergoing caesarean section, does chlorhexidine skin preparation reduce surgical site infection compared with povidone-iodine?

Ten minutes later, Bushra turned her laptop around. “Sir, Cochrane has already done it. Thirteen trials. Chlorhexidine probably slightly reduces surgical site infection, risk ratio 0.72 [12]. We are too late.”

“Perhaps. Look at the search date in the methods. July 2019. The authors also found two ongoing trials. Search PubMed and the trial registers for trials published since then, and check PROSPERO for anyone already updating it. If several new trials have appeared, an update with a registered protocol is a real contribution. If not, you have just saved yourselves six months, and that is also a result.”

Bushra looked at the screen for a long time. “Three steps need two people,” she said finally. “Sumaira, you screen first. I will screen separately. We do not look at each other’s decisions.”

“And when you disagree?” I asked.

Both of them looked at Zunaira. She smiled and nodded.

“Research is not rocket science,” I said as they packed up. “It is a skill, and skills can be taught. Tonight two rivals learnt the most important skill of a systematic review: working independently, and then honestly together.”

Key Takeaways

A systematic review answers one focused question with a registered protocol, a repeatable search, independent screening and extraction by two reviewers, and a risk of bias assessment. A meta-analysis is the optional statistical step that pools the results.
Check PROSPERO and the Cochrane Library before you start, register your protocol before screening, and report with the PRISMA 2020 checklist and flow diagram.
In a forest plot, each square is a study (size shows weight), each line its confidence interval, and the diamond the pooled result. If the diamond does not touch the line of no effect, the pooled result is significant.
I² shows how much the studies disagree; high heterogeneity is a question to explain, not to ignore. Funnel plots look for missing small studies but are unreliable with fewer than 10 studies.
Do not pool studies with different outcomes or definitions, and remember that biased trials give a biased diamond. Rate certainty with GRADE and appraise published reviews with AMSTAR 2.
Choose the design from the question: describe or compare, assign the exposure or not, and select on exposure or outcome. A meta-analysis is not a shortcut; many published ones are redundant, and editors know it.

References:

Liggins GC, Howie RN. A controlled trial of antepartum glucocorticoid treatment for prevention of the respiratory distress syndrome in premature infants. Pediatrics. 1972;50(4):515-25.
Crowley P, Chalmers I, Keirse MJ. The effects of corticosteroid administration before preterm delivery: an overview of the evidence from controlled trials. Br J Obstet Gynaecol. 1990;97(1):11-25.
McGoldrick E, Stewart F, Parker R, Dalziel SR. Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth. Cochrane Database Syst Rev. 2020;12(12):CD004454.
Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.5 [Internet]. Cochrane; 2024. Available from: https://www.cochrane.org/handbook
Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-60.
Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-34.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-6.
Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485-514.
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.
Hadiati DR, Hakimi M, Nurdiati DS, Masuzawa Y, da Silva Lopes K, Ota E. Skin preparation for preventing infection following caesarean section. Cochrane Database Syst Rev. 2020;6(6):CD007462.

Rate this article

How useful was it for you?