Grading Evidence

When a new treatment works as well as a standard one, it looks pretty good. But wait. In psychiatry, the placebo is often potent enough to overpower standard treatments. Think of all the negative SSRIs trials. Unless a placebo was around to ensure the reliability of the trial, we are left with doubt. Case in point: SAMe consistently equals antidepressants, but largely failed against placebo.

Big problem (3-4): Most trials in psychiatry.

Small problem (1-2): If the outcome is objective (eg, lab testing) or the disorder has a low placebo response (eg, tardive dyskinesia, schizophrenia, OCD, bipolar mania), or the treatment they are testing surpassed the control.

A quality trial states the primary outcome in advance. If it is not significant (p < 0.05), the trial is negative.

Suspect trials don’t name a primary outcome, and instead list a scatter plot of many outcomes. The more outcomes they test, the more likely a false-positive result, rendering the p-value useless. This kind of “p-hacking” or “data fishing” makes the results about as reliable as an observational study.

You can adjust for multiple outcomes with stricter cut-off for the p value. The most conservative method (the Bonferroni correction) divides the p cut-off (0.05) by the number of outcomes (eg, trials with 2 outcomes need a p value < 0.025). Other methods take into account the fact that many outcomes are correlated (eg, if a patient scores high on one depression scale they’re likely to score high on another).

Really big problem (make it uncontrolled): There’s enough (> 50%) negative outcomes to make the trial inconclusive. Rate it as an uncontrolled case series (2/10) and subtract no more.

Big problem (3-4): There’s enough (30-50%) negative outcomes in the trial to make the conclusion suspect.

Small problem (1-2): The majority of the outcomes are positive, even if the primary outcome missed the mark; or the p values are small enough to survive the Bonferroni correction.

Small trials are more likely to turn up false results. That includes positive and negative results, but only the false-positives are likely to get published.

The ideal size depends on how potent the treatment is (effect size) and how much certainty (power) you need.

If you don’t know the effect size, use 0.5, which is where the average psychiatric treatment falls. For a 0.5 effect, this chart suggests we’re 70% confident in the trial’s conclusions if the sample is at least 100 (Serdar CC et al, 2020).

This holds up if we’re testing two arms, like lithium vs. placebo, but we’ll need a larger trial to test more treatments (eg, a sample of 200 instead of 100 if testing 4 treatments instead of 2). Studies that use uneven allocation also need larger samples (eg, putting fewer patients in the placebo group).

Big problem (3): Sample size is less than 40 (or less than half the ideal size for 70% power).

Small problem (1-2): Sample size is more than 40 (or more than half the ideal size for 70% power).

Studies can blind various parties to the treatment: patients, their clinicians, the raters, and even an external board that monitors the study. Blinding is particularly important with subjective outcomes, as it is for the rating scales in most psychiatric studies.

Blinding is not possible with some treatments (psychotherapy, TMS, diet). In that case, we need a fair comparison, like a sham (fake) lightbox, or supportive psychotherapy that balances out the human interaction common to all therapies. Unfair and unreliable comparisons include wait list control and treatment as usual.

Another problem is functional unblinding. Ideally, no one can tell who got the placebo and who got the med. This ideal is rarely met. In depression, the accurate guess rate is 60% for SSRIs, 70-80% for esketamine, and >90% for psychedelics.

Big problem (2): For example, if no one was blind to the treatment, or if a wait list control was compared to psychotherapy.

Small problem (1): For example, a single-blind medication trial.

Cross-over trials allow investigators to enroll fewer subjects without compromising power, but they are less reliable than “parallel-group” trials:

• Carry over effects from previous treatment
• Fluctuations in the course of illness invalidate the results, so cross-over is only useful for chronic, stable diseases

Subtract 2-3 depending on how likely these problems were to influence the results.

This rating depends on 1) Percent of drop outs (< 20% ideal), 2) How evenly they were distributed between placebo/med, and 3) How they were handled. Your final score (1-4) is a judgment call based on how likely they affected the conclusion.

People drop out of trials for a reason. They are likely the more severe cases, or the ones who get worse (or no better) in the trial. Leaving them out of the final analysis biases the results. If they did not use an “intent to treat analysis,” downgrade the trial.

• Ideal: Full intent-to-treat analysis
• Good enough: Modified intent-to-treat analysis
• Problematic: “completer,” “complete case,” or “available case” analyses (evaluating only participants who complete the study), or “per protocol” analyses (only participants who completed all key study procedures)

Even when dropouts are accounted for, sometimes their sheer number makes the trial too uncertain. An arbitrary cut-off for the ideal is < 20% drop outs. Long-term trials and those involving unstable patients like borderline or substance use disorders will often cross that threshold.

• Ideal: Less than 20% of subjects dropout after randomization
• Small problem: 20-30%
• Big problem: Over 30% (or over 20% and grossly uneven between two groups)

People drop out of trials for a reason. They are likely the more severe cases, or the ones who get worse (or no better) in the trial. Leaving them out of the final analysis will bias the results. Look for “intent to treat analysis” and downgrade the trial if they didn’t do this and it likely affected the results.

There are various ways to analyze dropouts, including “last-observation-carried-forward” (carries last data point forward) and the more conservative “multiple imputation,” (averages multiple estimates for the missing value to better reflect uncertainty).

Who tried to enroll in the trial? Were any treatments stopped or changed before they entered it? These questions might reveal biases that favor the treatment under investigation. Psychedelic trials tend to attract people who’ve taken the drug before (with good experiences). If an antidepressant trial required subjects to stop other antidepressants before entering the trial, they may come in with withdrawal symptoms that favor the new antidepressant,

Even when dropouts are accounted for, sometimes their sheer number makes the trial too uncertain. An arbitrary cut-off is > 20% drop outs, but some trials are bound to cross that threshold – such as long-term studies and those involving unstable patients like borderline or substance use disorders – so we may need to take what we can get.

We want to avoid the risk that investigators will shunt certain types of patients into the active drug group, or that they can guess who got randomized to which group. Ideally, the randomization is removed from the investigator’s office, such as by phone, web, or through a pharmacist or centralized service (allocation concealment). Coded containers and SNOSE (sequentially numbered envelopes) also works.

If the randomization looks suspect and the baseline characteristics of the subjects are skewed, downgrade the trial.

1. How plausible are the results, judging from earlier studies? If the study concludes that stimulants treat psychosis, we’re going to need a lot more proof to move the needle.
2. How relevant is the outcome? In a study of alcohol use disorder, did they measure cravings in a laboratory setting or actual use in the real world?
3. How representative is the population? Did they draw from a sample of treatment seeking patients? What are the exclusion criteria? Sometimes when we lack data relevant to our patient we have to borrow from similar populations, like using a child ADHD trial to for an adult patient.