Background The management of depression in primary care is a significant issue for health services worldwide. ‘Collaborative care’ interventions are effective, but little is known about which aspects of these complex interventions are essential.
Aims To use meta-regressionto identify ‘active ingredients’ in collaborative care models for depression in primary care.
Method Studies were identified using systematic searches of electronic databases. The content of collaborative care interventions was coded, together with outcome data on antidepressant use and depressive symptoms. Meta-regression was used to examine relationships between intervention content and outcomes.
Results There was no significant predictor of the effect of collaborative care on antidepressant use. Key predictors of depressive symptom outcomes included systematic identification of patients, professional background of staff and specialist supervision.
Conclusions Meta-regression may be usefulin examining‘active ingredients’ in complex interventions in mental health.
Depression is prevalent in primary care, but current management is suboptimal (Simon & Von Korff, 1995). There is increasing evidence of the effectiveness of ‘ collaborative care’ (Gilbody et al, 2003), a multifaceted organisational intervention involving new staff and ways of working (Von Korff & Goldberg, 2001). However, collaborative care interventions vary in content and intensity, and it is unclear which aspects are crucial determinants of effectiveness (the ‘active ingredients’). Most of the current collaborative care literature derives from the USA, and designing collaborative care interventions for use in other settings requires an understanding of these ‘active ingredients’.
Collaborative care is an example of a complex intervention, involving a number of separate mechanisms, where the ‘active ingredient’ is difficult to specify (Campbell et al, 2000). If different collaborative care interventions vary in their inclusion of ‘active ingredients’, then this should lead to significant variation in outcomes. Such variation in outcomes in a meta-analysis is described as statistical heterogeneity. Meta-regression is a method used to explore statistical heterogeneity in meta-analysis (Sutton et al, 1998; Thompson & Higgins, 2002).
A phased approach to the development of complex interventions has been proposed (Campbell et al, 2000). Modelling of complex interventions, where the ‘active ingredients’ are explored, is a critical step in the phased model prior to further trials. However, there are relatively few examples of the phased model in the literature (Bradley et al, 1999; Campbell et al, 2000; Medical Research Council, 2000; Loeb, 2002) and a lack of consensus as to the optimal modelling methods.
The authors are developing and testing a collaborative care intervention in the UK using the phased approach, and used meta-regression to examine the relationship between the content of collaborative care interventions and outcomes, to identify ‘active ingredients’ and thus assist in the design of a UK collaborative care model for the care of depression.
We based our meta-regression on a systematic review. A published systematic review of organisational interventions in primary care mental health completed by S.G. was used as the initial source of studies (Gilbody et al, 2003); this review included collaborative care as well as other types of organisational interventions used to improve the management of depression. Searches included Medline, EMBASE, CINAHL, PsycINFO, the Cochrane Library and the Database of Abstracts of Reviews of Effectiveness (DARE), and were run from the inception date of each database to 2003. We updated the comprehensive search strategies from this review (without language restriction) to June 2004 to find recent studies, and then conducted a second update to November 2005 (Fig. 1). Details of the exact search methods and a table of excluded studies are available from the authors upon request. From this comprehensive database, we then identified the subset of collaborative care studies.
The population of interest was patients with depressive symptoms or diagnosed depressive disorders in primary care settings. Primary care was defined as the provision of medical care by professionals who provide first contact and ongoing care to patients, regardless of the patient’s age, gender or presenting problem.
Although we have published a broad typology of models of quality improvement which includes collaborative care (Bower & Gilbody, 2005), developing precise inclusion criteria for such complex interventions is more problematic, because by definition it is not clear a priori which mechanisms have to be in place in order to define an intervention as ‘ collaborative care’. Therefore, any definition is potentially arbitrary, reflected by published reviews of collaborative care that disagree on which studies and interventions are included and excluded (Von Korff & Goldberg, 2001; Gilbody et al, 2003; Bijl et al, 2004).
The purpose of the study was to examine the relationship between variation in the interventions within collaborative care studies, and outcomes. Therefore, we used a broad definition, and defined ‘collaborative care’ as a multifaceted organisational intervention, which could include a number of components:
the introduction of a new role (case manager) into primary care, to assist in the management of patients with depression through structured and systematic delivery of interventions;
the introduction of mechanisms to foster closer liaison between primary care clinicians and mental health specialists (including case managers) around individual patient care;
the introduction of mechanisms to collect and share information on the progress of individual patients.
We excluded educational and training interventions and the provision of brief psychological therapy where these were the sole intervention and were not supported by other enhancements of care outlined above. The full list of studies is given in Table 1.
Content of collaborative care
We initially tested published coding schemes relating to quality improvement (Weingarten et al, 2002; Bero et al, 2006), but these lacked the detail to capture the specific issues of relevance to collaborative care. Therefore, a basic coding frame was developed on the basis of the ‘prototypical’ collaborative care model, described in terms of the three roles potentially involved in patient care: primary care provider, mental health specialist and case manager (Katon et al, 2001b). Variables were created relating to the professional background of each worker and additional intervention-specific training. These codes were then supplemented by variables describing the potential interprofessional relationships (e.g. specialist supervision of the case manager, and case manager feedback of information to the primary care provider). Because professional–patient contact within collaborative care is focused on the case manager–patient relationship, we added variables relating to the intensity and nature of that contact. Finally, we added three variables related to the characteristics of the patients and study setting (see Appendix).
After piloting the data extraction among the team, data from each study were extracted by two different members of the research team working independently. There was no formal measurement of reliability, but disagreements were few and were resolved by discussion. Owing to inconsistent reporting of data in the published papers we were only able to extract comprehensive data on 8 of the original 27 variables (see Table 2).
Concealment of allocation is the quality attribute with the best evidence for an association with outcomes (Schultz & Grimes, 2002), and we extracted data on concealment to test whether outcomes were related to study quality.
Collaborative care interventions often seek to improve adherence to antidepressant medication, and the first outcome measure was changes in measures of antidepressant use. Most studies reported data in dichotomous form, e.g. the proportion of patients taking antidepressants or meeting standardised guidelines for antidepressant use.
The second outcome measure was reduction in depressive symptoms. A wide variety of outcomes were reported at a number of different time points. Because the meta-regression required as large a sample of studies as possible for reliable analysis (Thompson & Higgins, 2002), we restricted the analysis to short-term outcomes (approximately 6 months after randomisation), as these outcomes were by far the most frequently reported. Where alternative measures of depressive outcomes were reported within the same study, the data extracted were chosen on the basis of an a priori decision rule which extracted any identified primary outcome first, and then prioritised observer-rated scales over self-report measures where available.
We extracted all measures of anti-depressant use as dichotomous outcomes, analysed using odds ratios. Measures of depressive symptoms included a mix of dichotomous and continuous outcomes. We translated continuous measures to a standardised effect size, i.e. the mean of the intervention group minus the mean of the control group, divided by the pooled standard deviation. We translated outcomes reported as dichotomous variables to standardised effect sizes using the logit transformation (Lipsey & Wilson, 2001). In 5 of 62 (8%) comparisons, missing data (e.g. standard deviations) were imputed from other relevant studies, in line with accepted practice (Furukawa et al, 2006).
Previous reviews have identified that unit of analysis errors are common in the evaluation of collaborative care (Gilbody et al, 2003), making studies more susceptible to type 1 errors. We identified all studies using cluster randomisation and where necessary adjusted the precision of these studies in the meta-analysis using methods recommended by the Effective Practice and Organisation of Care (EPOC) group of the Cochrane Collaboration (Bero et al, 2006) and assuming an intraclass correlation of 0.02. The effects of adjustment for clustering were examined in a sensitivity analysis using intraclass correlations of 0.00 and 0.05 (Donner & Klar, 2002).
Analyses were conducted in Stata version 8 for Windows, using the metan and metareg macros. The initial meta-analyses used random effects modelling (Sutton et al, 1998) to provide an overall pooled measure of effect of collaborative care on the two outcomes. However, the main focus of the analysis was on heterogeneity. Heterogeneity was measured using the I2 statistic, which estimates the percentage of total variation across studies that can be attributed to heterogeneity rather than chance. As a guide, I2 values of 25% may be considered low, 50% moderate and 75% high (Higgins et al, 2003).
The main analysis used random effects meta-regression, which provided estimates of the relationships between eight intervention content variables and the two outcomes. The permutation test was used to calculate P values (using 1000 Monte Carlo simulations) and to reduce the chance of spurious false-positive findings (Higgins & Thompson, 2004). The amount of heterogeneity explained by the intervention content variables was examined by reductions in the I2 statistic. Initial univariate analyses (using a criterion of significance of P<0.10) were followed by estimation of a multivariate model. The multivariate model was not based on any automated selection procedure, but involved examination of a number of candidate models involving different combinations of variables. The final model was chosen on the basis of the greatest reduction in heterogeneity. A secondary meta-regression provided an estimate of the relationships between the two outcomes (i.e. whether antidepressant use predicted depressive symptoms).
We identified 28 published studies of collaborative care interventions with outcome data on antidepressant use and 34 studies with outcome data on depressive symptoms (Table 1). Intervention content variables are summarised in Table 2.
We found a positive effect of collaborative care on antidepressant use (odds ratio 1.92, 95% CI 1.54–2.39; Fig. 2) and depressive outcomes (standardised mean difference 0.24, 95% CI 0.17–0.32; Fig. 3). The I2 estimates of inconsistency were 80% and 54% respectively.
Analyses of the effects of intervention content variables are shown in Tables 3 and 4. There was insufficient variability in quality of allocation concealment, as most studies were rated as ‘not clear’, and this variable was not used as a covariate in the final analysis.
None of the intervention content variables was significantly associated with anti-depressant use, and no multivariate model was estimated. Three intervention content variables predicted improvement in depressive symptoms: recruitment by systematic identification (P=0.061), case managers having a specific mental health background (P=0.004) and provision of regular supervision for case managers (P=0.033), which reduced the overall heterogeneity I2 from 54% to 48% and 43 to 49% respectively.
In multivariate analysis, four intervention content variables produced the most robust meta-regression in relation to depressive symptom outcomes. The analysis indicated that non-US studies (P=0.038), recruiting through systematic identification of patients (P=0.081) and using case managers having a specific mental health background (P=0.027) who received regular supervision (P=0.055) were more effective. The combination of these four covariates reduced the overall heterogeneity to 36% (low to moderate between study heterogeneity). The inclusion of ‘ setting’ (which was not statistically significant in the univariate analyses) reflects the fact that the multivariate analysis accounts for both the relationships between each intervention content variable and the outcome, and the relationships between intervention content variables (Tabachnick & Fidell, 2001).
The meta-regression of the relationships between antidepressant use and depressive symptoms showed a positive association (β coefficient 0.20, 95% CI 0.02–0.38, P=0.028; Fig. 4).
The results of these analyses were not substantively influenced by the sensitivity analysis using estimates of intracluster correlations of 0.00 and 0.05.
Overall, the analysis showed an interesting pattern of results. No variable predicted variation in relation to our first outcome, antidepressant use. However, the study did identify several predictors of the second outcome, depressive symptoms. Furthermore, antidepressant use did predict depressive symptom outcomes, which suggests that effects of collaborative care on the latter may be mediated through changes in the former. However, it is not clear whether this association would remain significant if the antidepressant use variables were analysed alongside the other intervention content variables. Clearly the causal pathways between intervention content variables, intermediate outcomes (such as antidepressant use) and final outcomes (such as depressive symptoms) are potentially complex, and analytic techniques such as path analysis might be useful in examining these relationships further.
If the associations between intervention content variables and depressive symptom outcomes are robust, they have interesting implications for the design of collaborative care interventions. For example, the use of case managers with a mental health background and regular specialist supervision both predict outcomes, which suggests that expertise is important. This may have implications for the involvement of the new paraprofessional graduate workers in collaborative care models (Whitty & Gilbody, 2005).
Clearly the meta-regression cannot determine the process by which expertise has its influence. This may relate to specific technical skills, such as knowledge of anti-depressants or the effective use of psychotherapeutic techniques, or may reflect non-specific skills, such as the ability to engage with patients or to work effectively in collaboration with other professionals. Exploration of this issue might benefit from qualitative research on the nature of patient–professional and interprofessional contact in collaborative care, and the influence of context and organisational variables (Weaver et al, 2003).
However, models of care which require personnel with significant expertise are likely to be more difficult to implement in some contexts, which may limit their usefulness, reflecting the potential tension between ‘ efficacy’ as demonstrated in trials and ‘ effectiveness’ in routine contexts. Also, models using expert personnel may be more costly, which raises issues about trade-offs between effectiveness and cost that need to be considered when designing collaborative care interventions.
Limitations of the systematic review
As a complex intervention, collaborative care defies simple definition. Our decisions about inclusion and exclusion were informed by our previous conceptual work (Bower & Gilbody, 2005), but we took a liberal approach to inclusion precisely because the study focused on the degree to which variability in collaborative care models influenced outcomes. Clearly the inclusion or exclusion of particular studies may have important implications, and thus our findings should be considered exploratory rather than definitive. It should also be noted that most studies were conducted in the USA and the results may not generalise to other contexts. Setting was a significant predictor in the multivariate analysis.
The validity of the coding scheme used to extract data on the interventions has not been confirmed. As noted previously, there were problems of inconsistent reporting and missing data in the published studies. A significant proportion of intervention content variables could not be included as they were not reported consistently, and it is unlikely that it would have been possible to extract data on many additional issues. However, it remains possible that other variables might be more effective predictors than those included in our analyses.
The difficulties encountered in deriving a full description of the interventions echoes traditional problems with poor reporting in randomised trials. There may be a case for adopting a more standardised approach to the reporting of the content of complex interventions (equivalent to CONSORT (Consolidated Standards of Reporting Trials; Moher et al, 2001) and QUOROM (Quality of Reporting Metaanalyses; Moher et al, 1999) in order to overcome these problems. The proliferation of web-based journal archives for the presentation of data outside the word limits of the paper-based journals provides an appropriate platform. However, determining the appropriate content and structure of such standardised reports would be challenging, given the potential range of processes involved in complex interventions.
Limitations of the meta-regression technique
The technique of meta-regression has several limitations (Thompson & Higgins, 2002). The analysis represents an observational association only, because meta-regression across trials does not have the benefits of randomisation. Equally, statistical power to detect useful associations using meta-regression is limited by (among other things) the number of available studies (Lambert et al, 2002). Outliers may have a large influence, particularly in the context of a limited sample size. The multivariate model described earlier was found to be sensitive to the particular variables included in the analysis. It should also be noted that the analysis will not be able to detect ‘ active ingredients’ that are necessary but do not vary between interventions. Furthermore, it is possible that with certain variables, such as the number of case management sessions, the relationship with average numbers of sessions across trials may not be the same as the relationship within trials. Only individual patient data analysis could overcome this ‘ ecological fallacy’ (Thompson & Higgins, 2002).
Finally, the analyses were not controlled for quality criteria. The a priori quality criterion (concealment of allocation) showed little variation, as the majority of studies failed to report this adequately. However, it is not clear whether inadequate reporting of concealment always reflects inadequate methods (Soares et al, 2004; Pildal et al, 2005).
Alternatives to meta-regression in the analysis of complex interventions
The controversy over fidelity to assertive community treatment and outcomes (Fiander et al, 2003) indicates that the identification and measurement of ‘active ingredients’ in mental health interventions has important implications for both research and service provision (Marshall & Creed, 2000). It is therefore critical to consider the optimal methods of identifying ‘ active ingredients’. Our study has shown that the use of meta-regression is feasible but has limitations. The key issue is how well meta-regression compares with the available alternatives, which include clinical expertise, qualitative work, theoretical models and ‘ dismantling’ or ‘factorial’ trials.
Clinical expertise is a potentially useful source of hypotheses, and rigorous qualitative work is ideally suited to capture the complexity of care processes, and is especially useful at exploring the perspectives of stakeholders and illuminating context (Weaver et al, 2003; Marshall et al, 2004). However, it is unclear whether patients and professionals can reliably identify ‘active ingredients’. Acknowledgement of the limitations of clinical expertise in identifying causal mechanisms is fundamental to evidence-based medicine, and patients will presumably face many of the same challenges as professionals. Insights from theoretical models are another useful source, but few theoretical models within mental health services research are so well validated that they provide a comprehensive description of ‘active ingredients’, and complex mental health issues such as depression will have many competing theories. Although theory is a necessary aspect of the development of a complex intervention, it will rarely be sufficient.
Dismantling and factorial studies test different combinations of ingredients within a randomised comparison. Relevant examples exist in the collaborative care literature. For example, a recent study compared outcomes in patients randomised to a depression care programme (including systematic follow-up) and systematic follow-up alone. There was no difference in outcomes, suggesting that systematic follow-up is critical (Vergouwen et al, 2005). The advantage of such designs is that randomisation is preserved, allowing causal inference. However, the use of such costly designs to identify ‘active ingredients’ may not always be the optimal use of limited research resources.
Clearly comparisons of the different methods are required, and the intervention development currently being conducted by the authors also includes qualitative work which can be compared with the findings of the meta-regression. It is likely that complex interventions will increasingly be required to improve patient care within mental health, and the evaluation of such interventions raises particular challenges. Although there are potential problems with the application of meta-regression, we conclude that the technique has potential in developing useful insights into the active ingredients in complex interventions in mental health, and thus assist in the design and evaluation of future interventions.
The study was implemented in the context of research led by D.R. (funded by the Medical Research Council) in which P.B. and S.G. are grant holders. P.B. and J.F. are funded through the Department of Health, but the views expressed in this paper are those of the authors alone. P.B. thanks John Cape for useful input into the paper.
- Received February 28, 2006.
- Revision received May 10, 2006.
- Accepted June 2, 2006.
- © 2006 Royal College of Psychiatrists