Electronic Letters to:
|
|
eLetters: Electronic letters published:
-
![[Read eLetter]](/icons/misc/sectionDOWN.gif)
Flawed analysis of North Carolina Trial
- Matthew Hotopf, Graham Dunn, Gareth Owen, and Rachel Churchill (2 May 2007)
-
Looking Closely at the Evidence from the North Carolina Study of Outpatient Commitment
- Jeffrey W. Swanson, Marvin Swartz, M.D., Professor of Psychiatry, Duke University School of Medicine (8 May 2007)
|
|
|||
|
Matthew Hotopf, Psychiatrist Institute of Psychiatry, Graham Dunn, Gareth Owen, and Rachel Churchill
Send letter to journal:
m.hotopf{at}iop.kcl.ac.uk Matthew Hotopf, et al.
|
Swanson et al’s reanalyses of the results of the North Carolina trial, published in a series of articles including the current one, are becoming increasingly influential in current debates about mental health legislation in UK. Our recent systematic review (Churchill et al. 2007), which included these articles, demonstrated that there was no robust evidence to indicate that community treatment orders are associated either significant benefit or harm. The secondary analyses performed by Swanson and colleagues are, we believe, misleading for two reasons. 1. Based on everyone in the trial, the intention to treat (ITT) effect of randomisation to an OPC was of a modest and non-significant reduction in violence (risk difference of 4.5%). Now, this overall ITT effect of OPCs is a weighted average of the ITT effects in the two sub- groups of participants defined by their post-randomisation management (those who received short-term OPCs and those who eventually received long -term OPCs). These two subgroups would exist in the control arm had they been placed on OPCs. Assuming that there was no benefit in those who received the short-term OPCs (i.e. risk difference = 0) the results of Swanson et al’s study (together with some simple arithmetic) suggest that the reduction in violence in those with long-term OPCs would be 12.4% (workings available from authors). However, even if thought clinically significant, this finding would still not be statistically significant because the overall ITT effect was not significant (assuming a zero ITT effect in those receiving short-term OPCs implies that a test of the hypothesis concerning those receiving long-term OPCs is equivalent to the test for the overall ITT effect). The only way in which there could have been a beneficial effect in those receiving long-term OPCs is if the effects in those receiving short-term OPCs were actually detrimental (i.e. increased the rate of violence). It is improbable that they would be, and in policy terms it would be unacceptable to impose OPCs in the knowledge that they would cause harm to those in whom they are only applied for a short time period, especially as this is a larger group in the North Carolina trial. 2. A post hoc comparison of the outcomes in groups defined by management decisions or patient behaviour following randomisation is potentially subject to selection effects (hidden confounding). That this is in fact the case is illustrated by the results of other subgroup analyses by the same research group ((Swartz et al. 1999) figure 1). The group destined to be on long-term OPC have a better clinical outcome in the first 1-2 months. In other words there is evidence that the group destined to receive long-term OPCs have a favourable clinical profile before the OPC is renewed. We believe that it is likely that long-term OPCs will only be contemplated under certain circumstances, such as whether the short-term OPC has apparently made a difference to the patient’s condition. Those who have intractable problems or in whom a short-term OPC has failed to make any change, may not have their OPC renewed. The investigators responsible for the North Carolina trial accomplished one of the most extraordinary trials ever performed, and as such deserve enormous praise. However the results described in these and similar secondary analyses are, we believe, flawed and misleading, and should not be taken as evidence for a beneficial effect of OPC. We made a similar point following the publication of the original trial (Szmukler and Hotopf, 2001). The trial data are best interpreted using the main intention to treat analyses which show no evidence of benefit or harm. Professor Matthew Hotopf Professor Graham Dunn Dr Gareth Owen Dr Rachel Churchill References Churchill, R., Owen, G., Singh, S., & Hotopf, M. 2007, International experiences of using community treatment orders, Department of Health, London. Swartz, M. S., Swanson, J. W., Wagner, H. R., Burns, B. J., Hiday, V. A., & Borum, R. 1999, "Can Involuntary Outpatient Commitment Reduce Hospital Recidivism?: Findings From a Randomized Trial With Severely Mentally Ill Individuals", American Journal of Psychiatry, vol. 156, no. 12, pp. 1968-1975. Szmukler G and Hotopf M Effectiveness of involuntary outpatient commitment Am J Psych, 2001 158 653-4 |
|||
|
|
|||
|
Jeffrey W. Swanson, Associate Professor in Psychiatry and Behavioral Sciences Duke University School of Medicine, Marvin Swartz, M.D., Professor of Psychiatry, Duke University School of Medicine
Send letter to journal:
jeffrey.swanson{at}duke.edu Jeffrey W. Swanson, et al.
|
We are pleased that the results of our North Carolina study of outpatient commitment (OPC) continue to animate scientific discussion of this controversial topic in mental health policy and law. At a time when Community Treatment Orders (CTOs) are under renewed consideration in the UK, a careful reexamination of the evidence for the effectiveness of these legal instruments is well worth the trouble. Hotopf and colleagues have criticized two separate published reports from our OPC study, on hospital readmissions (Swartz et al., 1999) and violence (Swanson et al., 2000). These critics bring essentially one indictment – post-randomization selection bias – in their attempt to impugn both papers. In what follows, we address the criticisms separately as pertaining to each of the articles in turn. Hospitalization analysis: Hotopf and colleagues, here and elsewhere (Szmukler & Hotopf, 2001), omit the fact that we reported a statistically significant experimental result for the ITT sample. For any month during the study year, the randomly-assigned OPC group had a lower risk of readmission than did the control group (OR=0.64, 95% CI=0.46–0.88, p<0.01; Swartz et al., 1999, p.1972.) Our analysis also revealed that this positive outcome -- i.e. reduced risk of rehospitalization -- was concentrated heavily among participants who received extended, renewed court-ordered treatment in combination with frequent outpatient services; and that it worked best for patients with psychotic disorders. Still, the fact remains: the straight-up RCT result, obtained from standard repeated-measures logistic regression analysis, was strong enough to achieve statistical significance for the experimental group as a whole. That fact has been widely ignored by our critics, on the one side, and by strong advocates of OPC, on the other; both have tended to emphasize our post-randomization analyses exclusively -- either to debunk the study, or to use it to promote long-term OPC -- while largely ignoring the caution we advised. The number of days of OPC that participants actually received varied from 1 to 365 in a continuous (though bimodal) distribution, which we dichotomized at 6 months for the analysis. About one third of the OPC group had their court orders expire very early in the study -- during the first or second month -- and more of these individuals were soon rehospitalized than those remaining on OPC, which explains the early separation of the lines in the figures from Swartz et al (1999). Could short-term OPC actually be harmful? We did not find statistically significant differences in outcomes between the control group and participants who received only short-term OPC. However, we think it is plausible that brief OPC could have unintended adverse consequences. Consider, by way of analogy, medical patients who stop their 10-day antibiotic regimen after 2 days and then relapse with a more intractable infection than they started with. Likewise, one might imagine involuntary psychiatric patients getting a short-lived court order -- long enough to feel coerced, but not for any benefit of treatment to take hold. It is not unreasonable to think such patients might become more resistant to treatment than ever, and with measurably worse outcomes. Violence analysis: With respect to our violence study, Hotopf and colleagues have taken some trouble to make essentially the same point that we ourselves stated very clearly in the article: “. . . [T]he study found no significant difference in the prospective rate of violence between the two randomly assigned groups: 32.3% in the OPC group v. 36.8% in the control group (Fisher's exact test, one-tailed: P=0.292; two-tailed: P=0.567).” Critics of OPC policy wish we had left it at that. But it would have been misleading to do so, for two reasons. First, these null results could not be generalized to violent patients in the real world, because we excluded all seriously violent patients (n=64) from random assignment, as required by the court and institutional policy. We did include these (historically violent) patients in an observational arm of the study, because we were aware that such patients may represent precisely the people that clinicians, family members, policy-makers, and the public are often most concerned about when it comes to the question of whether OPC can work. Some important policy questions are simply not amenable to study by a randomized clinical trial. To try to answer such questions, we do the best we can with available methods -- in this case a hybrid of an efficacy trial and real-world effectiveness study -- and we qualify the results accordingly. Second, and fortuitously for our study (if not for the patients themselves), following initial assignment to OPC the historically violent patients ended up receiving highly variable periods of court-ordered treatment. This natural variability in the real-world application of OPC gave us the rare opportunity to compare rates of repeat violence for those with brief vs. extended OPC. Those with longer periods of court-ordered treatment had significantly lower rates of violence over the study year. We think it would have been irresponsible on our part not to report this finding, with appropriate methodological caveats. Hotopf and colleagues have attempted to recalculate the post- randomization effect for longer-term OPC in what they refer to as our ITT sample, rather than the sample we actually used; they report that the result is not significant. Unfortunately, their calculation excludes the historically violent subgroup, making it not very interesting and not quite correct. Moreover, unlike the hospitalization study, our violence study did not actually have a true ITT sample to analyze in the first place. Participants who withdrew from the study (20.9% by 12 months) were not around to be assessed for violence at follow-up. This is another limitation, which we acknowledge. As to the possibility of selection bias affecting the post- randomization analysis, we clearly stated in the paper: “Amount of time on OPC was not random or controlled experimentally, but varied as clinicians applied the legal criteria for renewal of OPC orders. Potentially, this could have lead to a biased conclusion (i.e. attributing a positive intervention effect to subjects who might have been less violent anyway because of pre-existing lower risk).” However, we think that favorable selection bias was quite unlikely, for the following reasons: Our study protocol, the legal criteria for OPC in North Carolina, and patterns in our own data suggest just the opposite. Participating mental health centers agreed in advance to systematically review each expiring court order and file a petition to renew if the patient continued to meet legal criteria for OPC. We sent reminders to the clinical teams before each order was due to expire, setting out the legal criteria. And we tracked whether it was done. Consequently, patients with a history of treatment noncompliance turned out to be more than twice as likely to have their orders renewed during the year (40.0% v. 18.75%). This may have amounted to a selection bias, but one favoring a negative finding, making it harder -- not easier -- to show an effect for OPC. In the end, some will be persuaded by these results and others will not. As researchers, we take no advocacy position with respect to the policy of OPC or CTOs. Our goal has been to help build a base of evidence to inform the policy debate. Clearly, however, this is an area that warrants careful reflection as well as further research in different populations and jurisdictions. Jeffrey Swanson, Ph.D. and Marvin Swartz, M.D. Duke University School of Medicine Durham, North Carolina, USA References Swanson, J. W., Swartz, M. S., Wagner, H. R., Burns, B. J., Borum, R. Hiday, V. A. 2000. Involuntary out-patient commitment and reduction of violent behaviour in persons with severe mental illness. British Journal of Psychiatry, 176, 324-331 Swartz, M. S., Swanson, J. W., Wagner, H. R., Burns, B. J., Hiday, V. A., Borum, R. (1999) Can involuntary outpatient commitment reduce hospital recidivism? Findings from a randomised trial in severely mentally ill individuals. American Journal of Psychiatry, 156, 1968-1975. Szmukler, G., and Hotopf, M. (2001) Effectiveness of involuntary outpatient commitment (letter). American Journal of Psychiatry, 158, 653- 4. Author information: Jeffrey Swanson, Ph.D. Associate Professor Department of Psychiatry & Behavioral Sciences Duke University School of Medicine DUMC Box 3071 Durham, NC 27710 Telephone: (919)682-4827 Fax: (919) 682-1907 E-mail: jeffrey.swanson@duke.edu Marvin Swartz, M.D. Professor and Head, Division of Social and Community Psychiatry Department of Psychiatry and Behavioral Sciences Duke University School of Medicine DUMC Box 3173 Durham, NC 27710 Telephone: (919) 684-8676 Fax: (919) 681-7504 E-mail: swart001@mc.duke.edu Declaration of interest: Neither Dr. Swanson nor Dr. Swartz have received fees or grants from, were ever employed by, served as consultants for, had shared ownership in, or any close relationship with, any organisation whose interests, financial or otherwise, may be affected by publication of the letter. |
|||