Quasi-Experimental Studies: How to Evaluate an Intervention When You Cannot Randomise

The Counselling Desk

Dr. Junaid Rashid came in with a circular from the Medical Superintendent’s office folded into his shirt pocket.

“Sir, from the first of next month the hospital is opening an adherence counselling desk in the medicine OPD. A trained staff nurse will sit with every hypertensive patient for ten minutes: what each tablet does, why it must not be stopped when the headache goes, how to use a pill box.” He put the circular on the table. “And the MS has asked me to tell him, in six months, whether it works.”

“That is a research question with a client already waiting,” I said. “What is your plan?”

“Simple. I measure adherence in one hundred patients in October, before the desk opens. Then I measure it again in the same patients in April. If adherence goes up, the desk works.”

Dr. Hammad Ali could not hold himself back. “Randomise it, Sir! Half the patients go to the desk, half do not. Then it is a proper trial. My territory.”

Junaid shook his head before I could answer. “I already tried that in my head. The desk is for everyone. The MS will not allow half the OPD to be refused counselling that the hospital is paying for. And even if he did, my patients sit on the same benches for two hours. The ones who are counselled will tell the ones who are not. By lunchtime, half my control group will have heard the whole lecture.”

“That is called contamination,” I said, “and you have just explained, in one minute, why a great deal of real health service research cannot be randomised. A new protocol in a ward, a training programme for nurses, a checklist in the theatre, a policy for the whole district. The decision is made by the hospital, not by the researcher, and it is made for everyone at once. For questions like that, we use a quasi-experimental design.”

What “Quasi” Means

I wrote it on the board.

In a quasi-experimental study, an intervention is deliberately introduced, but chance does not decide who receives it.

“Last week,” I said, “we met quasi-random allocation: odd and even dates pretending to be randomisation. Today is different. A quasi-experimental study does not pretend at all. Everyone knows the groups were not created by chance, and the whole design is about protecting the conclusion anyway.”

“So it sits between an observational study and a trial,” said Dr. Sumaira Talib.

“Exactly between them. Like a cohort study, the groups may differ in ways you never measured. Like a trial, the intervention is introduced on purpose, at a known point in time. Harris and colleagues described a ladder of these designs, from weak to strong [1]. Tonight we climb it, starting from Junaid’s plan at the bottom rung.”

The Bottom Rung: One Group, Before and After

“Junaid, suppose in April adherence has gone up from 52% to 68%. You walk into the MS’s office with that result. What else, apart from the desk, could explain it?”

He thought. “If the price of medicines changes. Or if there is a campaign on television about blood pressure.”

“Good. That is called history: anything else that happens in the world between your two measurements. Six months is a long time in Pakistan. A free-medicine scheme can be announced, a famous person can have a stroke on the news, Ramadan can arrive and change when everyone takes their tablets. Your design cannot tell any of these apart from the desk.”

Dr. Bushra Fatima added one. “The patients who come back in April are not all the patients from October. The ones who stopped their tablets may have stopped coming to the OPD as well.”

“Attrition, and it works in the worst direction. The non-adherent disappear and the adherent remain, so adherence rises even if the desk does nothing.”

“And they will have answered the same questions twice,” said Sumaira. “The second time, they know which answer the doctor wants.”

“The testing effect. Closely related to the Hawthorne effect we met in the cohort session: people behave and answer differently when they know they are being studied.”

“Who will measure adherence in April, Junaid?”

“The counselling nurse, I thought. She will know the patients.”

“She also wants her desk to succeed. If a different person, or a different method, measures in April, any change may come from the measurement and not the patient. That is instrumentation.”

The Quietest Threat

Dr. Zunaira Malik had been reading the circular upside down from across the table.

“Sir, it says here that patients with systolic pressure above 160 will be sent to the desk first.” She looked up. “If we choose the patients because their pressure is very high today, will it not come down next time anyway?”

The room went quiet. Junaid turned the circular round and read the line himself.

“That,” I said, “is regression to the mean, and it is the threat that fools even consultants. Think of a batsman who scores a duck. The selectors send him to a new batting coach. In the next match he makes thirty. Was it the coach? Perhaps. But a batsman who scored zero was very likely to score more next time anyway, because zero was partly bad luck. Blood pressure is the same. A reading of 175 on one morning is partly the patient’s true pressure and partly that morning’s rush, anxiety and missed breakfast. Select people for being extreme, measure them again, and on average they move back towards the middle, with or without any intervention [2].”

“So if the desk only counsels the worst patients,” Zunaira said slowly, “it will look successful even if it does nothing.”

“Yes. The defence is to take all eligible patients, not only the extreme ones, and to have a comparison group selected in the same way.”

I drew the summary on the board.

Threat	How it could appear in the OPD	Defence
History	A free-medicine scheme or media campaign during the six months	Comparison group measured over the same months
Maturation	Newly diagnosed patients settle into a routine on their own	Comparison group; enrol established patients separately
Testing	Patients learn the “right” answers to the adherence questions	Same tool in both groups; an objective measure such as pill count where possible
Instrumentation	The counselling nurse measures the outcome herself	An independent data collector using one fixed tool
Regression to the mean	Only patients with very high pressure are sent to the desk	Enrol all eligible patients consecutively, in both groups
Attrition	Non-adherent patients stop coming to follow-up	Record and report every loss; compare those lost with those followed

The Second Rung: Add a Comparison Group

“Junaid, how many medicine units run the OPD?”

“Two. Medicine Unit I and Medicine Unit II, on different days.”

“And will the desk open for both on the first of the month?”

He checked. “Unit I first. Unit II in January, when the second nurse finishes her training.”

“Then the MS has handed you a comparison group without knowing it. Measure adherence in both units in October, and again in December, before Unit II gets the desk. This is the non-equivalent control group design. ‘Non-equivalent’ because the two units were not created by chance, and you must say so.”

I put numbers on the board to show how the analysis works.

	Before (October)	After (December)	Change
Unit I (desk)	52%	68%	+16 points
Unit II (no desk yet)	50%	58%	+8 points

“Look at Unit II. Adherence rose by eight points without any desk. That is history, testing and everything else working together. So the honest estimate of the desk’s effect is not sixteen points. It is the difference between the two changes: sixteen minus eight, about eight points.”

“The difference of the differences,” said Junaid.

“That is literally its name: difference-in-differences. It assumes that, without the desk, both units would have changed by the same amount. So you also compare the two units at baseline: age, duration of hypertension, number of tablets, education. If they differ, the statistical adjustment is something we will cover in the confounding session.”

Hammad was still thinking about his territory. “Sir, if the MS had let us decide by lottery which unit gets the desk first, would that be a trial?”

“A cluster randomised trial, yes, though with only two units, chance has very little room to balance anything. When a programme is rolled out to many wards or many basic health units in stages, and the order is decided randomly, it becomes a stepped-wedge design. Remember the idea; it is how many health programmes can be evaluated fairly.”

The Third Rung: Interrupted Time Series

Sumaira had been tapping her pen. “Sir, in surgery we have registers going back years. Our infection control nurse keeps a monthly chart of wound infections on the ward. When chlorhexidine skin preparation came in last year, everyone said infections fell. Is that research?”

“It can be, if you treat the chart properly. What was the line doing before chlorhexidine?”

“I never looked. Everyone only compared the month before with the month after.”

“Then nobody knows whether infections were already falling. This design is called an interrupted time series. You need many measurements before the change and many after, usually monthly. One month on each side is a before-and-after study. A year of monthly points on each side lets you see the trend. Then you ask two questions: did the level jump at the moment of the change, and did the slope change afterwards?”

I drew two lines on the board. In the first, infections were falling steadily for a year before chlorhexidine and simply kept falling at the same rate. In the second, the line was flat for a year and dropped sharply in the month the new practice started.

“Both charts would look like success in a simple before and after comparison. Only the second has earned it.”

“Junaid, your OPD register records blood pressure at every visit. You could plot, month by month, the proportion of hypertensive patients whose pressure is controlled, for a year before the desk and a year after. The analysis is called segmented regression, and it has to account for things like seasons. Pressures rise in winter, and Ramadan changes how tablets are taken. Bernal and colleagues have written a very clear tutorial on it [3]. You do not need to master the statistics tonight. You need to collect the right data from the start.”

Writing It Up Honestly

Dr. Hassan Raza had stayed quiet until now. “Sir, just tell me the format. What do I write as the study design?”

“Write what you actually did, in full. Not ‘randomised controlled trial’, not just ‘quasi-experimental’. For Junaid: a quasi-experimental study with a non-equivalent control group, pre-test and post-test design. A reviewer should know the design from the title and the first line of the methods.”

“And is there a checklist, like CONSORT?”

“For non-randomised evaluations of behavioural and public health interventions, there is the TREND statement [4]. It asks you to describe the intervention, how the groups were formed and how you dealt with differences between them. And when a systematic review later includes your study, its authors will judge its risk of bias with a tool called ROBINS-I [5]. Write your methods as if someone with that tool will read every line, because someone will.”

“Does it still need the ERC?” Bushra asked.

“Yes. And remember the ICMJE definition from last week. It covers any study that prospectively assigns people or groups to an intervention, with or without a control group [6]. If the researcher assigns the intervention, a quasi-experimental study is a clinical trial in the eyes of most journals and should be registered before it starts. Here the hospital introduced the desk and Junaid is designing its evaluation, so he should discuss registration with the ERC at the start, not after the results come in. If the study compares groups, the hypothesis is non-directional, as we did in the objectives session.”

Junaid’s Revised Plan

Junaid was quiet for a while, writing. Then he read out his new plan, and I put it on the board.

Element	First plan	Revised plan
Design	One group, before and after	Non-equivalent control group (Unit I with desk, Unit II before its desk opens), pre-test and post-test
Patients	Those sent to the desk for systolic pressure above 160	All eligible hypertensive patients, enrolled consecutively in both units
Outcome measured by	The counselling nurse	A trained data collector not attached to the desk, using one fixed adherence tool
Timing	October and April	October and December in both units, before Unit II gets the desk
Analysis	Before compared with after	Change in Unit I compared with change in Unit II, with baseline differences checked
Extra strength	None	Monthly blood pressure control from the OPD register, a year before and after (interrupted time series)
Reporting	Not mentioned	TREND statement

“My main study stays the same,” he said. “The frequency of non-adherence in our OPD. This is the MS’s question. But now, when he asks me whether the desk works, I will not just say yes. I will be able to say how much, and how sure I am.”

“That is the difference between an audit and a study,” I said.

Hammad leaned over to look at the table. “Sir, it is still not as strong as my trial.”

“No,” I said. “But your trial will tell us about one pressure setting in one operating theatre. Junaid’s study will tell the MS whether to pay a nurse’s salary for the next ten years. Every design has its place. Research is not rocket science. It is a skill, and skills can be taught. Part of the skill is choosing the strongest design the situation allows, and being honest about what it cannot do.”

Key Takeaways

• A quasi-experimental study introduces an intervention deliberately, but without randomisation. It suits hospital protocols, training programmes and policies that are introduced for everyone at once.
• The one-group before-and-after design is the weakest. History, maturation, testing, instrumentation, attrition and regression to the mean can all produce a change without any real effect.
• Do not select only extreme patients for the intervention; their next readings will drift towards the mean on their own.
• Adding a comparison group measured over the same period allows a difference-in-differences estimate, which is far more honest than a simple before and after.
• An interrupted time series, with many monthly points before and after the change, shows whether the level or the trend really changed.
• Name the design fully, report it with the TREND statement, get ERC approval, and discuss registration if the intervention is assigned prospectively.

References:

1. Harris AD, McGregor JC, Perencevich EN, Furuno JP, Zhu J, Peterson DE, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc. 2006;13(1):16-23.
2. Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: what it is and how to deal with it. Int J Epidemiol. 2005;34(1):215-20.
3. Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46(1):348-55.
4. Des Jarlais DC, Lyles C, Crepaz N; TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. 2004;94(3):361-6.
5. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.
6. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals: clinical trial registration. Available from: https://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html