critically appraising the evidence

On completion of the search all items retrieved have to be reviewed and the appropriate material identified.

Of  all the items retrieved one study was of interferon beta-1b.  The other dealt with other forms of treatment and was thus discarded.  The beta-interferon study indicates that a course of interferon can reduce the rate of relapses by one-third in patients who have had  2 or more acute exacerbations in the past 2 years.  Side-effects were frequent, but mostly only had "nuisance-value".

Reference
The IFNB Multiple Sclerosis Study Group and the University of British Columbia MS/MRI Analysis Group. Interferon beta-1b in the treatmaent of multiple sclerosis: final outcome of the randomized controlled trial.  Neurology 1995; 45:1277-85

The validity of the evidence  is determined by  critically appraising it.  This is done by asking the following  questions:

1. Is the evidence valid; i.e. How close is it to the truth?
2. Is this evidence important, i.e. Can it be potentially useful to clinicians?

There is no rule as to the sequence in which the above has to be answered. Many clinicians will try the 2^nd question  first. If their answer to it is negative they will immediately discard that study and go on to the next one.

To illustrate the process of critical appraisal we will use "diagnostic test" as an example. It has to be remembered that the process can safely be applied to any of the Topics listed in Table 2

The validity of the evidence about a particular diagnostic test is determined by asking a number of questions:

1. Was there an independent, blind comparison with a reference  (”gold”) standard diagnosis?
2. Was the diagnostic test evaluated in an appropriate spectrum of patients (like those in whom it would be used in practice)?
3. Was the reference standard applied regardless of the diagnostic test result?

These questions can be applied to individual studies, but also to a systematic review (overview) of several studies of the same diagnostic test for the same disease or disorder.

With regard to the third question, when patients have a negative diagnostic test, it is sometimes felt to forego application  of the reference standard. When the reference standard is invasive it might even be felt inappropriate to do so. To this end many investigators use  a reference standard for patients not having the target disease or disorder  in which they don’t suffer any adverse health outcome during a long follow-up on no definitive treatment.

Should the report being analyzed fail to meet one or more of the criteria set out above, it needs to be considered whether it has a fatal flaw that might render its conclusions invalid. If this happen it means, "Search on for more evidence!"

If on the other hand the report passes the validity check, you can move on to the applicability check

Deciding whether or not a report is important / applicable the clinician is taking into account both his own clinical expertise as well as the best external evidence. The former is the prior assessment of diagnostic possibilities and the latter the ability of the test to distinguish between patients with and without the disease or disorder in question.

The pretest probabilities are derived from the clinician’s own accumulated clinical expertise specific to his area of practice. The result is that pretest probabilities may vary widely between as well as  within countries and within the different levels of care.

Can the evidence be applied to clinical practice?
Once the evidence has been found, critically appraised and deemed valid as well as important. The next, and final step, is incorporating it into the care of patients.

Applying the evidence
The results of the study is discussed with Mr Moretti.  It is  agreed that the evidence is there for interferon beta-1b to be beneficial.  With Mr Moretti's MS relapses not being so frequent it does not warrant  considering him for treatment.  Should he, however, require the use of interferon in future, the price of therapy  might have gone down.  Or some other course of treatment might even be available.

In order to decide on the integration of the evidence there are three questions to answer.

1. Is the diagnostic test available, affordable, accurate and precise in the setting in which it is to be used?
2. Is it possible to generate a clinically sensible estimate  of the patient’s pretest probability:
• From practice data?
• From personal experience?
• From the report itself?
• From clinical speculation?
3. Will the resulting post-test probabilities affect your management and help your patient?
• Could it move you across a test-treatment  threshold?
• Would the patient be willing to be a partner in carrying it out?
• Would the consequences of the test help  the patients reach their goals in all this?

Further Reading

1. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-based  Medicine - How to Practice and Teach EBM. Churchill Livingstone 1997
2. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ 1996; 312:71-2
3. Covell 09-Feb-2006tice: are they bieng met? Ann Intern Med 1985; 103: 596-9
4. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomised control trials and recommendations of clinical experts. JAMA 1992; 268:240-8
5. Davidoff F; Haynes B, Sackett D, Smith R. Evidence based medicine: a new journal to help doctors identify the information they need. BMJ 1995; 310:1085-6
6. Ramsey PG, Carline JD, Inui TS et al. Changes over time in the knowledge base of practicing internists. JAMA 1991; 266:1103-7
7. Evans CE, Haynes RB, Birkett NJ et al. Does a mailed continuing education program improve clinician performance? Results of a randomised trial in antihypertensive care. JAMA 1986; 255:501-4
8. Davis DA, Thompson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995; 274:700-5
9. Sibley JC, Sackett DL, Neufeld V et al. A randomised trial of continuing medical education. N Engl J Med 1982; 306:511-5
10. Shin JH, Haynes RB, Johnston ME. Effect of problem-based, self-directed undergraduate education on life-long learning. Can Med Assoc J 1993; 148:969-76
11. Oxman A, Guyatt GH. The science of reviewing research. Ann NY Acad Sci 1993; 703:125-34
12. Dickersin K, Sherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994; 309:1286-91
13. Fullerton-Smith I. How members of the Cochrane Collaboration prepare and maintain systematic reviews of the effects of health care. Evidence-Based Medicine 1995; 1:7-8
14. Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectivenes of CME. A review of 50 randomised controlled trials. JAMA 1992; 268:1111-7
15. Johnston ME, Langton KB, Haynes RB. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med 1994; 120:135-42
16. Systematic Reviews. Synthesis of Best Evidence for Health Care Decisions.  Edited: C Mulrow & D Cook. American College of Physicians, 1998