Introduction to Evidence-Based Medicine

David Stewart, MD

January 8, 2024

The Need for Evidence-Based Medicine

Physicians and other medical providers have long been among the most educated medical providers in the United States. Decades of medical research have driven improvements in the efficacy and quality of care.

While key areas of medical practice are well-established, numerous controversies exist. This discussion refers to mainstream controversies for which there are various approaches deemed medically reasonable within the broader standard of care, with practices often varying by region or training. It is important to emphasize that these controversies involve conditions for which different approaches have been described as medically acceptable and which have generally been reported to achieve adequate outcomes. It in no way refers to fringe views, such as those of vaccine deniers, climate change deniers, and others.

Wide variation exists in medical practice. Whereas these differences sometimes reflect experience and adaptation to individual patient needs, they can also reflect preferences and dogma. Irrational variation is associated with increased costs and lower quality.[1]

A study at Stanford University evaluated the rationale reported for 1,150 clinical decisions made by pediatric orthopedic surgeons in 2018. The authors reported:

“Out of 1150 total decisions, the most frequent decisions were follow-up scheduling, followed by bracing prescription/removal. The most common decision rationales were ‘First principles’ (n = 310, 27.0%) and ‘Experience/anecdote’ (n = 253, 22.0%). Only 17.8% of decisions were attributed to scientific studies, with 7.3% based on studies specific to the decision. As high as 34.6% of surgical intervention decisions were based on scientific studies, while only 10.4% of follow-up scheduling decisions were made with studies in mind.”[2]

The authors acknowledge that “many common conditions and clinical decisions faced daily in pediatric orthopedic clinics have not been studied,” but also note that the available evidence is broader than reflected in decision-making.

A similar study of clinical decisions of ten Harvard-affiliated pediatric cardiologists, also entitled “Deciding without Data,” studied the physician-reported rationale for 1188 clinical decisions made by ten pediatric cardiologists over 7.5 working days.[3] The authors reported: “Almost 80% of decisions were deemed by the physicians to have no basis in any prior published data and fewer than 3% of decisions were based on a study specific to the question at hand.”

In both studies, physician decision-making occurred within broad standards of care. Notwithstanding the need for further research in many areas, the available evidence exceeds its clinical utilization. Further emphasis on evidence-based medicine and clinical pathways can improve clinical quality and resource utilization.

AAOS Pediatric Orthopedic Guidelines

Since the start of evidence-based medicine initiatives in 2008, the American Academy of Orthopedic Surgeons (AAOS) lists clinical program guidelines or appropriate use criteria for only three pediatric conditions: developmental dysplasia of the hip, supracondylar humerus fractures, diaphyseal (shaft) femur fractures. Clavicle fractures is also listed, but the result is directed overwhelmingly towards adults. Only one full page addresses clavicle fractures in adolescents.[4] Even of the reportedly complete guidelines, many key clinical questions remain. The AAOS’ 2020 “Supplement to the Treatment of Pediatric Diaphyseal Femur Fracture” notes that key recommendations were removed from the 2009 guidelines due to the lack of evidence:[5]

“We are unable to recommend for or against early spica casting for children age six
months to five years with a diaphyseal femur fracture with greater than 2 cm of
shortening.
“We are unable to recommend for or against patient weight as a criterion for the
use of spica casting in children age six months to five years with a diaphyseal
femur fracture.
“We are unable to recommend for or against using any specific degree of
angulation or rotation as a criterion for altering the treatment plan when using the
spica cast in children six months to five years of age.
“We are unable to recommend for or against removal of surgical implants from
asymptomatic patients after treatment of diaphyseal femur fractures.
“We are unable to recommend for or against outpatient physical therapy to
improve function after treatment pediatric diaphyseal femur fractures.
“We are unable to recommend for or against the use of locked versus non-locked
plates for fixation of pediatric femur fractures.”

Controversies exist in each of these areas, with research of lesser levels of evidence demonstrating conflicting or disputed findings.

Levels of Evidence

The Journal of Bone and Joint Surgery’s Levels of Evidence define the standards for each category, with Level I evidence being the most robust and Level V the weakest. Some subjective element persists. There may be a tendency for some studies to overstate their level of evidence notwithstanding few study participants, excessive exclusions, and methodological difficulties which would appear to warrant downgrading.

A major change in the Level of Evidence is that while expert opinion, which was designated as Level V evidence in early iterations,[6] has been dropped from recent updates and replaced with mechanism-based reasoning. “Expert opinion” has been shown to be substantially fallible and is not privileged in any way under contemporary evidence-based schema. Even experts must pose “mechanism-based reasoning,” which involves rational explanation by known phenomena and are subject to critique. No more can experts simply proclaim facts by virtue of their personal authority. This change reflects mounting evidence demonstrating expert opinion to be limited and prone to bias.

The Curse of Expertise

In The Intelligence Trap: Why Smart People Make Dumb Mistakes (W.W. Norton 2021), David Robson noted that rationality is a different dimension than intelligence, and that highly educated people are just as susceptible to bias as others:

“The upshot, according to [Yale Law School researcher Dan] Kahan and other scientists studying motivated reasoning, is that smart people do not apply their superior intelligence fairly, but instead use it ‘opportunistically’ to promote their own interests and protect the beliefs that are most important to their identities. Intelligence can be a tool for propaganda rather than truth-seeking.” (pp. 54-55)

Robson describes the “curse of expertise:”

“The illusion of expertise may make you more close-minded. Victor Ottati at Loyola University in Chicago has shown that priming people to feel knowledgeable means that they were ‘less likely to seek or listen to the views of others who disagreed with them’’’... Experts may “overestimate their own knowledge” and fail to listen to other views or to update to different circumstances. (p. 71)
“Expertise…can also come with costly sacrifices. One is flexibility: the expert may lean so heavily on existing behavioral schemas that they struggle to cope with change… A second sacrifice may be an eye for detail…When fallible, gist-based processing is combined with overconfidence and ‘earned dogmatism,’ it gives us one final form of the intelligence trap.” (pp. 74-75)

Foundations of Evidence Based Medicine

John Ioannidis at Stanford University, widely recognized as the father of evidence-based medicine, wrote a landmark 2005 article entitled “Why Most Published Research Findings are False”[7] and a follow-up 2014 article entitled “How to Make More Published Research True.”[8] These articles are obligatory for close reading by those who want to perform quality research, as well as those who wish to better understand and interpret published research.

In “Why Most Published Research Findings are False,” Ioannidis observed that even well-intended authors introduce bias into research in “design, data, analysis, and presentation,” that “tend to produce research findings when they should not be produced.” He presents a mathematical model demonstrating based on the risk of alpha-error (risk of erroneously finding a relationship that does not actually exist), beta-error (risk of not finding a relationship that actually exists), and bias that the threshold for even 50% accuracy of published research findings is high and that the majority of published research falls below this standard. He wrote:

“It can be proven that most claimed research findings are false.
“Several methodologists have pointed out that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05.”
Bias is “the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.”
Bias including which factors to analyze, which to ignore, and how to model can drastically impact findings.
“In the described framework, a PPV exceeding 50% is quite difficult to get."

From mathematical modeling, Ioannidis arrives at key corollaries:

Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

Among the findings of “How to Make More Published Research True,”Ioannidis wrote: “Currently, many published research findings are false or exaggerated, and an estimated 85% of research resources are wasted.” He notes the perverse incentives introduced by university systems which reward the publication of research, with little regard to quality or reproducibility. He recommends, in part:

“Practices that have improved credibility and efficiency in specific fields may be transplanted to others which would benefit from them…
“Modifications need to be made in the reward system for science, affecting the exchange rates for currencies (e.g., publications and grants) and purchased academic goods (e.g., promotion and other academic or administrative power) and introducing currencies that are better aligned with translatable and reproducible research."

Whereas some findings are of high-quality, some research is of lower quality with different studies posing discrepant or contradictory results. Common limitations include retrospective studies, small numbers of patients, and methodological problems. While useful findings can often be gleaned from such research, research with lower Levels of Evidence warrants less confidence than more robust studies.

References

[1] Shea, Kevin G. MD. Strategies and Tools to Enhance Patient Safety: HROs, HEROs, and Safety Culture. Journal of Pediatric Orthopaedics 40():p S30-S32, July 2020. | DOI: 10.1097/BPO.0000000000001500

[2] Nathan K, Uzosike M, Sanchez U, et al. Deciding without data: clinical decision-making in pediatric orthopedic surgery. Int J Qual Health Care. 2020;32(10):658-662. doi:10.1093/intqhc/mzaa119

[3] Darst JR, Newburger JW, Resch S, Rathod RH, Lock JE. Deciding without data. Congenit Heart Dis. 2010;5(4):339-342. doi:10.1111/j.1747-0803.2010.00433.x

[4] "Treatment of Clavicle Fractures: Evidence-Based Clinical Practice Guidelines." American Academy of Orthopedic Surgeons, December 2, 2022. https://www.aaos.org/globalassets/quality-and-practice-resources/clavicle-fractures/clavicle-fractures-cpg.pdf

[5] "Supplement to the Treatment of Pediatric Diaphyseal Femur Fracture." American Academy of Orthopedic Surgeons, 2020. https://www.aaos.org/globalassets/quality-and-practice-resources/pdff/pdff-2020-eappendix.pdf

[6] Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128(1):305-310. doi:10.1097/PRS.0b013e318219c171

[7] Ioannidis JP. Why most published research findings are false [published correction appears in PLoS Med. 2022 Aug 25;19(8):e1004085]. PLoS Med. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124

[8] Ioannidis JP. How to make more published research true. PLoS Med. 2014;11(10):e1001747. Published 2014 Oct 21. doi:10.1371/journal.pmed.1001747