REVIEW ARTICLE |
National Primary Care Research and Development Centre, University of Manchester
Department of Health Sciences, University of York
Department of Nursing, Midwifery and Health Visiting, University of Manchester
Department of Health Sciences, University of Leicester, UK
Correspondence: Dr Peter Bower, National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL, UK. Email: peter.bower{at}manchester.ac.uk
|
|
|---|
Aims To use meta-regressionto identify active ingredients in collaborative care models for depression in primary care.
Method Studies were identified using systematic searches of electronic databases. The content of collaborative care interventions was coded, together with outcome data on antidepressant use and depressive symptoms. Meta-regression was used to examine relationships between intervention content and outcomes.
Results There was no significant predictor of the effect of collaborative care on antidepressant use. Key predictors of depressive symptom outcomes included systematic identification of patients, professional background of staff and specialist supervision.
Conclusions Meta-regression may be usefulin examiningactive ingredients in complex interventions in mental health.
|
|
|---|
Collaborative care is an example of a complex intervention, involving a number of separate mechanisms, where the active ingredient is difficult to specify (Campbell et al, 2000). If different collaborative care interventions vary in their inclusion of active ingredients, then this should lead to significant variation in outcomes. Such variation in outcomes in a meta-analysis is described as statistical heterogeneity. Meta-regression is a method used to explore statistical heterogeneity in meta-analysis (Sutton et al, 1998; Thompson & Higgins, 2002).
A phased approach to the development of complex interventions has been proposed (Campbell et al, 2000). Modelling of complex interventions, where the active ingredients are explored, is a critical step in the phased model prior to further trials. However, there are relatively few examples of the phased model in the literature (Bradley et al, 1999; Campbell et al, 2000; Medical Research Council, 2000; Loeb, 2002) and a lack of consensus as to the optimal modelling methods.
The authors are developing and testing a collaborative care intervention in the UK using the phased approach, and used meta-regression to examine the relationship between the content of collaborative care interventions and outcomes, to identify active ingredients and thus assist in the design of a UK collaborative care model for the care of depression.
|
|
|---|
![]() View larger version (23K): [in a new window] [as a PowerPoint slide] |
Fig. 1 QUOROM (Quality of Reporting Meta-analyses) flow diagram.
|
Although we have published a broad typology of models of quality improvement which includes collaborative care (Bower & Gilbody, 2005), developing precise inclusion criteria for such complex interventions is more problematic, because by definition it is not clear a priori which mechanisms have to be in place in order to define an intervention as collaborative care. Therefore, any definition is potentially arbitrary, reflected by published reviews of collaborative care that disagree on which studies and interventions are included and excluded (Von Korff & Goldberg, 2001; Gilbody et al, 2003; Bijl et al, 2004).
The purpose of the study was to examine the relationship between variation in the interventions within collaborative care studies, and outcomes. Therefore, we used a broad definition, and defined collaborative care as a multifaceted organisational intervention, which could include a number of components:
We excluded educational and training interventions and the provision of brief psychological therapy where these were the sole intervention and were not supported by other enhancements of care outlined above. The full list of studies is given in Table 1.
|
View this table: [in a new window] | Table 1 Studies included in the review |
Data extraction
Content of collaborative care
We initially tested published coding schemes relating to quality
improvement (Weingarten et al,
2002; Bero et al,
2006), but these lacked the detail to capture the specific issues
of relevance to collaborative care. Therefore, a basic coding frame was
developed on the basis of the prototypical collaborative care
model, described in terms of the three roles potentially involved in patient
care: primary care provider, mental health specialist and case manager
(Katon et al,
2001b). Variables were created relating to the
professional background of each worker and additional intervention-specific
training. These codes were then supplemented by variables describing the
potential interprofessional relationships (e.g. specialist supervision of the
case manager, and case manager feedback of information to the primary care
provider). Because professional–patient contact within collaborative
care is focused on the case manager–patient relationship, we added
variables relating to the intensity and nature of that contact. Finally, we
added three variables related to the characteristics of the patients and study
setting (see Appendix).
After piloting the data extraction among the team, data from each study were extracted by two different members of the research team working independently. There was no formal measurement of reliability, but disagreements were few and were resolved by discussion. Owing to inconsistent reporting of data in the published papers we were only able to extract comprehensive data on 8 of the original 27 variables (see Table 2).
|
View this table: [in a new window] | Table 2 Intervention content variables (n=34) |
Concealment of allocation is the quality attribute with the best evidence for an association with outcomes (Schultz & Grimes, 2002), and we extracted data on concealment to test whether outcomes were related to study quality.
Intervention outcomes
Collaborative care interventions often seek to improve adherence to
antidepressant medication, and the first outcome measure was changes in
measures of antidepressant use. Most studies reported data in dichotomous
form, e.g. the proportion of patients taking antidepressants or meeting
standardised guidelines for antidepressant use.
The second outcome measure was reduction in depressive symptoms. A wide variety of outcomes were reported at a number of different time points. Because the meta-regression required as large a sample of studies as possible for reliable analysis (Thompson & Higgins, 2002), we restricted the analysis to short-term outcomes (approximately 6 months after randomisation), as these outcomes were by far the most frequently reported. Where alternative measures of depressive outcomes were reported within the same study, the data extracted were chosen on the basis of an a priori decision rule which extracted any identified primary outcome first, and then prioritised observer-rated scales over self-report measures where available.
We extracted all measures of anti-depressant use as dichotomous outcomes, analysed using odds ratios. Measures of depressive symptoms included a mix of dichotomous and continuous outcomes. We translated continuous measures to a standardised effect size, i.e. the mean of the intervention group minus the mean of the control group, divided by the pooled standard deviation. We translated outcomes reported as dichotomous variables to standardised effect sizes using the logit transformation (Lipsey & Wilson, 2001). In 5 of 62 (8%) comparisons, missing data (e.g. standard deviations) were imputed from other relevant studies, in line with accepted practice (Furukawa et al, 2006).
Previous reviews have identified that unit of analysis errors are common in the evaluation of collaborative care (Gilbody et al, 2003), making studies more susceptible to type 1 errors. We identified all studies using cluster randomisation and where necessary adjusted the precision of these studies in the meta-analysis using methods recommended by the Effective Practice and Organisation of Care (EPOC) group of the Cochrane Collaboration (Bero et al, 2006) and assuming an intraclass correlation of 0.02. The effects of adjustment for clustering were examined in a sensitivity analysis using intraclass correlations of 0.00 and 0.05 (Donner & Klar, 2002).
Analysis
Analyses were conducted in Stata version 8 for Windows, using the
metan and metareg macros. The initial meta-analyses used
random effects modelling (Sutton et
al, 1998) to provide an overall pooled measure of effect of
collaborative care on the two outcomes. However, the main focus of the
analysis was on heterogeneity. Heterogeneity was measured using the
I2 statistic, which estimates the percentage of total
variation across studies that can be attributed to heterogeneity rather than
chance. As a guide, I2 values of 25% may be considered
low, 50% moderate and 75% high (Higgins
et al, 2003).
The main analysis used random effects meta-regression, which provided estimates of the relationships between eight intervention content variables and the two outcomes. The permutation test was used to calculate P values (using 1000 Monte Carlo simulations) and to reduce the chance of spurious false-positive findings (Higgins & Thompson, 2004). The amount of heterogeneity explained by the intervention content variables was examined by reductions in the I2 statistic. Initial univariate analyses (using a criterion of significance of P<0.10) were followed by estimation of a multivariate model. The multivariate model was not based on any automated selection procedure, but involved examination of a number of candidate models involving different combinations of variables. The final model was chosen on the basis of the greatest reduction in heterogeneity. A secondary meta-regression provided an estimate of the relationships between the two outcomes (i.e. whether antidepressant use predicted depressive symptoms).
|
|
|---|
Meta-analysis
We found a positive effect of collaborative care on antidepressant use
(odds ratio 1.92, 95% CI 1.54–2.39;
Fig. 2) and depressive outcomes
(standardised mean difference 0.24, 95% CI 0.17–0.32;
Fig. 3). The
I2 estimates of inconsistency were 80% and 54%
respectively.
![]() View larger version (36K): [in a new window] [as a PowerPoint slide] |
Fig. 2 Meta-analysis of antidepressant use. Note: the Wells (2000) and Simon
(2004) studies involved two intervention groups compared against a single
control; to avoid double-counting the controls, the sample size and event rate
in the control were divided by 2. The Rost 2001 study data are only available
analysed in two subgroups, rather than as an overall analysis; in our analysis
these subgroups were treated as separate comparisons.
|
![]() View larger version (39K): [in a new window] [as a PowerPoint slide] |
Fig. 3 Meta-analysis of depressive symptoms (see note for
Fig. 2).
|
|
View this table: [in a new window] | Table 3 Univariate analysis of associations between intervention content variables and antidepressant use |
|
View this table: [in a new window] | Table 4 Univariate analysis of associations between intervention content variables and depressive symptoms |
None of the intervention content variables was significantly associated with anti-depressant use, and no multivariate model was estimated. Three intervention content variables predicted improvement in depressive symptoms: recruitment by systematic identification (P=0.061), case managers having a specific mental health background (P=0.004) and provision of regular supervision for case managers (P=0.033), which reduced the overall heterogeneity I2 from 54% to 48% and 43 to 49% respectively.
In multivariate analysis, four intervention content variables produced the most robust meta-regression in relation to depressive symptom outcomes. The analysis indicated that non-US studies (P=0.038), recruiting through systematic identification of patients (P=0.081) and using case managers having a specific mental health background (P=0.027) who received regular supervision (P=0.055) were more effective. The combination of these four covariates reduced the overall heterogeneity to 36% (low to moderate between study heterogeneity). The inclusion of setting (which was not statistically significant in the univariate analyses) reflects the fact that the multivariate analysis accounts for both the relationships between each intervention content variable and the outcome, and the relationships between intervention content variables (Tabachnick & Fidell, 2001).
The meta-regression of the relationships between antidepressant use and depressive symptoms showed a positive association (β coefficient 0.20, 95% CI 0.02–0.38, P=0.028; Fig. 4).
![]() View larger version (8K): [in a new window] [as a PowerPoint slide] |
Fig. 4 Relationship between antidepressant use outcomes and depressive symptoms
outcomes.
|
|
|
|---|
If the associations between intervention content variables and depressive symptom outcomes are robust, they have interesting implications for the design of collaborative care interventions. For example, the use of case managers with a mental health background and regular specialist supervision both predict outcomes, which suggests that expertise is important. This may have implications for the involvement of the new paraprofessional graduate workers in collaborative care models (Whitty & Gilbody, 2005).
Clearly the meta-regression cannot determine the process by which expertise has its influence. This may relate to specific technical skills, such as knowledge of anti-depressants or the effective use of psychotherapeutic techniques, or may reflect non-specific skills, such as the ability to engage with patients or to work effectively in collaboration with other professionals. Exploration of this issue might benefit from qualitative research on the nature of patient–professional and interprofessional contact in collaborative care, and the influence of context and organisational variables (Weaver et al, 2003).
However, models of care which require personnel with significant expertise are likely to be more difficult to implement in some contexts, which may limit their usefulness, reflecting the potential tension between efficacy as demonstrated in trials and effectiveness in routine contexts. Also, models using expert personnel may be more costly, which raises issues about trade-offs between effectiveness and cost that need to be considered when designing collaborative care interventions.
Limitations of the systematic review
As a complex intervention, collaborative care defies simple definition. Our
decisions about inclusion and exclusion were informed by our previous
conceptual work (Bower & Gilbody,
2005), but we took a liberal approach to inclusion precisely
because the study focused on the degree to which variability in collaborative
care models influenced outcomes. Clearly the inclusion or exclusion of
particular studies may have important implications, and thus our findings
should be considered exploratory rather than definitive. It should also be
noted that most studies were conducted in the USA and the results may not
generalise to other contexts. Setting was a significant predictor in the
multivariate analysis.
The validity of the coding scheme used to extract data on the interventions has not been confirmed. As noted previously, there were problems of inconsistent reporting and missing data in the published studies. A significant proportion of intervention content variables could not be included as they were not reported consistently, and it is unlikely that it would have been possible to extract data on many additional issues. However, it remains possible that other variables might be more effective predictors than those included in our analyses.
The difficulties encountered in deriving a full description of the interventions echoes traditional problems with poor reporting in randomised trials. There may be a case for adopting a more standardised approach to the reporting of the content of complex interventions (equivalent to CONSORT (Consolidated Standards of Reporting Trials; Moher et al, 2001) and QUOROM (Quality of Reporting Metaanalyses; Moher et al, 1999) in order to overcome these problems. The proliferation of web-based journal archives for the presentation of data outside the word limits of the paper-based journals provides an appropriate platform. However, determining the appropriate content and structure of such standardised reports would be challenging, given the potential range of processes involved in complex interventions.
Limitations of the meta-regression technique
The technique of meta-regression has several limitations
(Thompson & Higgins,
2002). The analysis represents an observational association only,
because meta-regression across trials does not have the benefits of
randomisation. Equally, statistical power to detect useful associations using
meta-regression is limited by (among other things) the number of available
studies (Lambert et al,
2002). Outliers may have a large influence, particularly in the
context of a limited sample size. The multivariate model described earlier was
found to be sensitive to the particular variables included in the analysis. It
should also be noted that the analysis will not be able to detect
active ingredients that are necessary but do not vary between
interventions. Furthermore, it is possible that with certain variables, such
as the number of case management sessions, the relationship with average
numbers of sessions across trials may not be the same as the relationship
within trials. Only individual patient data analysis could overcome this
ecological fallacy (Thompson
& Higgins, 2002).
Finally, the analyses were not controlled for quality criteria. The a priori quality criterion (concealment of allocation) showed little variation, as the majority of studies failed to report this adequately. However, it is not clear whether inadequate reporting of concealment always reflects inadequate methods (Soares et al, 2004; Pildal et al, 2005).
Alternatives to meta-regression in the analysis of complex interventions
The controversy over fidelity to assertive community treatment and outcomes
(Fiander et al, 2003)
indicates that the identification and measurement of active
ingredients in mental health interventions has important implications
for both research and service provision
(Marshall & Creed, 2000).
It is therefore critical to consider the optimal methods of identifying
active ingredients. Our study has shown that the use of
meta-regression is feasible but has limitations. The key issue is how well
meta-regression compares with the available alternatives, which include
clinical expertise, qualitative work, theoretical models and
dismantling or factorial trials.
Clinical expertise is a potentially useful source of hypotheses, and rigorous qualitative work is ideally suited to capture the complexity of care processes, and is especially useful at exploring the perspectives of stakeholders and illuminating context (Weaver et al, 2003; Marshall et al, 2004). However, it is unclear whether patients and professionals can reliably identify active ingredients. Acknowledgement of the limitations of clinical expertise in identifying causal mechanisms is fundamental to evidence-based medicine, and patients will presumably face many of the same challenges as professionals. Insights from theoretical models are another useful source, but few theoretical models within mental health services research are so well validated that they provide a comprehensive description of active ingredients, and complex mental health issues such as depression will have many competing theories. Although theory is a necessary aspect of the development of a complex intervention, it will rarely be sufficient.
Dismantling and factorial studies test different combinations of ingredients within a randomised comparison. Relevant examples exist in the collaborative care literature. For example, a recent study compared outcomes in patients randomised to a depression care programme (including systematic follow-up) and systematic follow-up alone. There was no difference in outcomes, suggesting that systematic follow-up is critical (Vergouwen et al, 2005). The advantage of such designs is that randomisation is preserved, allowing causal inference. However, the use of such costly designs to identify active ingredients may not always be the optimal use of limited research resources.
Clearly comparisons of the different methods are required, and the intervention development currently being conducted by the authors also includes qualitative work which can be compared with the findings of the meta-regression. It is likely that complex interventions will increasingly be required to improve patient care within mental health, and the evaluation of such interventions raises particular challenges. Although there are potential problems with the application of meta-regression, we conclude that the technique has potential in developing useful insights into the active ingredients in complex interventions in mental health, and thus assist in the design and evaluation of future interventions.
|
View this table: [in a new window] | Appendix Initial intervention content variables |
|
|
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
B. L. Rollman, B. H. Belnap, M. S. LeMenager, S. Mazumdar, P. R. Houck, P. J. Counihan, W. N. Kapoor, H. C. Schulberg, and C. F. Reynolds III Telephone-Delivered Collaborative Care for Treating Post-CABG Depression: A Randomized Controlled Trial JAMA, November 18, 2009; 302(19): 2095 - 2103. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. C. Stange A Science of Connectedness Ann. Fam. Med, September 1, 2009; 7(5): 387 - 395. [Full Text] [PDF] |
||||
![]() |
B. L. Rollman, B. H. Belnap, M. S. LeMenager, S. Mazumdar, H. C. Schulberg, and C. F. Reynolds III The Bypassing the Blues Treatment Protocol: Stepped Collaborative Care for Treating Post-CABG Depression Psychosom Med, February 1, 2009; 71(2): 217 - 230. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. van Orden, T. Hoffman, J. Haffmans, P. Spinhoven, and E. Hoencamp Collaborative Mental Health Care Versus Care as Usual in a Primary Care Setting: A Randomized Controlled Trial Psychiatr Serv, January 1, 2009; 60(1): 74 - 79. [Abstract] [Full Text] [PDF] |
||||
![]() |
I. Anderson, I. Ferrier, R. Baldwin, P. Cowen, L Howard, G Lewis, K Matthews, R. McAllister-Williams, R. Peveler, J Scott, et al. Evidence-based guidelines for treating depressive disorders with antidepressants: A revision of the 2000 British Association for Psychopharmacology guidelines J Psychopharmacol, June 1, 2008; 22(4): 343 - 396. [Abstract] [PDF] |
||||
![]() |
M. Timonen and T. Liukkonen Management of depression in adults BMJ, February 23, 2008; 336(7641): 435 - 439. [Full Text] [PDF] |
||||
![]() |
A. J. Mitchell Pooled Results From 38 Analyses of the Accuracy of Distress Thermometer and Other Ultra-Short Methods of Detecting Cancer-Related Mood Disorders J. Clin. Oncol., October 10, 2007; 25(29): 4670 - 4681. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. S. Wang, G. E. Simon, J. Avorn, F. Azocar, E. J. Ludman, J. McCulloch, M. Z. Petukhova, and R. C. Kessler Telephone Screening, Outreach, and Care Management for Depressed Workers and Impact on Clinical and Work Productivity Outcomes: A Randomized Controlled Trial JAMA, September 26, 2007; 298(12): 1401 - 1411. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||