Hostname: page-component-7c8c6479df-24hb2 Total loading time: 0 Render date: 2024-03-27T12:49:53.879Z Has data issue: false hasContentIssue false

The evidence–practice gap in specialist mental healthcare: Systematic review and meta-analysis of guideline implementation studies

Published online by Cambridge University Press:  02 January 2018

Francesca Girlanda*
Affiliation:
Department of Psychiatry ll, Ulm University, Ulm, Germany, and Department of Public Health and Community Medicine, Section of Psychiatry, University of Verona, Verona, Italy
Ines Fiedler
Affiliation:
Department of Psychiatry ll, Ulm University, Ulm, Germany
Thomas Becker
Affiliation:
Department of Psychiatry ll, Ulm University, Ulm, Germany
Corrado Barbui
Affiliation:
Department of Public Health and Community Medicine, Section of Psychiatry, University of Verona, Verona, Italy
Markus Koesters
Affiliation:
Department of Psychiatry ll, Ulm University, Ulm, Germany
*
F. Girlanda, Department of Psychiatry II, Ulm University, Ludwig-Heilmeyer-Str. 2, 89312 Günzburg, Germany. Email: francesca.girlanda@uni-ulm.de
Rights & Permissions [Opens in a new window]

Abstract

Background

Clinical practice guidelines are not easily implemented, leading to a gap between research synthesis and their use in routine care.

Aims

To summarise the evidence relating to the impact of guideline implementation on provider performance and patient outcomes in mental healthcare settings, and to explore the performance of different strategies for guideline implementation.

Method

A systematic review of randomised controlled trials, controlled clinical trials and before-and-after studies comparing guideline implementation strategies v. usual care, and different guideline implementation strategies, in patients with severe mental illness.

Results

In total, 19 studies met our inclusion criteria. The studies did not show a consistent positive effect of guideline implementation on provider performance, but a more consistent small to modest positive effect on patient outcomes.

Conclusions

Guideline implementation does not seem to have an impact on provider performance, nonetheless it may influence patient outcomes positively.

Type
Review Articles
Copyright
Copyright © The Royal College of Psychiatrists 2017 

Clinical practice guidelines have become a standard tool in the attempt to increase the quality of clinical care. Surprisingly, whereas the pathway from evidence generation to evidence synthesis and guideline development is highly developed and sophisticated, the pathway from evidence-based guidelines to evidence-based practice is much less developed. Reference Fretheim, Schünemann and Oxman1 Implementation methods range from simple interventions, such as dissemination of educational material, to more complex and multifaceted interventions, including tutorial and consultation sessions, use of treatment algorithms, reminder systems, audit and feedback and use of psychological theories to overcome obstacles. Reference Grimshaw, Thomas, MacLennan, Fraser, Ramsay and Vale2 Whether these implementation strategies are effective in terms of better provider performance and patient outcomes in severe mental illness has been researched in previous systematic reviews. Reference Weinmann, Koesters and Becker3Reference Girlanda, Fiedler, Ay, Barbui and Koesters5 Although these reviews found insufficient high-quality evidence to draw firm conclusions on the effects of implementation of psychiatric guidelines, they report modest effects towards an improvement of healthcare provider performance and the clinical conditions of the patients. Reference Weinmann, Koesters and Becker3Reference Girlanda, Fiedler, Ay, Barbui and Koesters5 In spite of these findings, guidelines are used to develop quality indicators for healthcare. These indicators usually measure adherence to guideline recommendation under the assumption that an increased adherence to an evidence-based guideline would subsequently lead to improved patient outcomes. Therefore, there is need for up-to-date evidence on guideline implementation in order to verify this assumption. Furthermore, it is relevant to understand how guidelines should be best implemented into practice. In order to shed light on these issues, the present systematic review updated previous research and summarised the evidence on the effects of guideline implementation on provider performance and patient outcomes.

Method

This review followed an a priori-defined protocol that was published on our institutional website in 2012. Reference Koesters, Girlanda, Ay, Cipriani and Barbui6 The protocol was developed following the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). Reference Liberati, Altman, Tetzlaff, Mulrow, Gøtzsche and Ioannidis7

Study inclusion criteria

Types of studies and participants

This systematic review included randomised controlled trials (RCTs), controlled clinical trials (CCTs) and before-and-after studies. We considered lack of a control condition an exclusion criterion. We included non-RCTs because guideline implementation is complex and often investigated in non-RCTs. Only including RCTs would therefore miss large parts of the evidence available. However, we only conducted the meta-analyses for the RCTs. Study participants were adults of either gender, with a primary diagnosis of schizophrenia or related psychotic disorder, unipolar and bipolar affective disorders and severe depression. As our main interest was the care of patients with severe mental illness, we only included studies with participants recruited and treated in specialist mental healthcare settings. We excluded studies in non-adult populations because of the different treatment process in children and adolescents, which may involve parents.

Types of intervention

We considered any active or passive guideline implementation strategy. For the purposes of this review, we defined guidelines as ‘systematically developed statements (or algorithms, flow charts or tables) prepared to assist decisions about appropriate healthcare for specific clinical circumstances’. Reference Barbui, Girlanda, Ay, Cipriani, Becker and Koesters4 An implementation strategy was defined as ‘any planned process and systematic introduction of guidelines with the aim of giving them a structural place in professional practice’. Reference Barbui, Girlanda, Ay, Cipriani, Becker and Koesters4 We classified the interventions for implementing guidelines according to a taxonomy developed by the Cochrane Effective Practice and Organisation of Care Review Group (EPOC). 8 The following comparisons were included: (a) guidelines implementation strategy v. usual care to examine the impact of guideline implementation on provider performance and patient outcomes; (b) guidelines implementation strategy A v. guidelines implementation strategy B to understand if effects can be augmented by enhanced implementation strategies.

Outcome measures

The primary outcome measure was provider performance, because clinical practice guidelines are developed to improve the performance of healthcare providers in accordance with the best available scientific evidence. The secondary outcome measures were patient outcomes, such as psychopathological symptoms, satisfaction with care, treatment adherence, attitude towards psychiatric medications, and quality of life, as defined by each of the studies.

We hypothesised that guideline implementation may have a positive impact on healthcare provider performance and that this would subsequently influence patient outcomes. Furthermore, we expected that outcomes would differ in different studies according to the characteristics and purposes of the guideline under scrutiny. Only one outcome for provider performance and one patient outcome for each study (where available) was selected for the analyses. The criterion used to select the outcomes was their strength to shed light on the impact of guideline implementation strategies in real clinical practice. Furthermore, we made the selection of the outcomes on the basis of their coherence with the other selected measures. With regard to the performance of healthcare providers we selected outcomes that clearly indicated the level of adherence to the implemented guideline (for example proportion of patients with polypharmacy). Psychopathological symptoms measured by validated scales were considered the primary indicator of the impact of the implementation process on patients (patient outcomes).

Search strategy for identification of studies

We performed a literature search of the following electronic databases: EMBASE, Ovid MEDLINE, PsycINFO, PSYNDEX, Cochrane Database of Systematic Reviews and Cochrane Central Register of Controlled Trials. The electronic search was run using the comprehensive search strategy listed in online supplement DS1 and was last updated in November 2015. Additionally, we screened the references of all included studies for further relevant literature. We did not apply language restrictions.

Data collection

We screened all titles and abstracts, and obtained full texts of potentially relevant papers. Working independently and in duplicate, two reviewers (F.G. and I.F.) read the papers and determined whether they met inclusion criteria. Two reviewers (F.G. and I.F.) carried out data extraction, using a standardised sheet. The information collected included study design, patient characteristics (such as diagnosis), provider characteristics (profession), setting of care, type of intervention (such as purpose of recommendation, guideline development process, implementation strategy) and outcome measures (process and patient outcomes). We solved disagreements by discussion. If no consensus could be reached, the final decision was taken together with a third reviewer (M.K.). For continuous outcomes we extracted the mean score at end-point (for RCTs), the mean change from baseline to end-point (for CCTs) and the mean score at baseline and end-point (for before-and-after studies), along with the standard deviation of these values and the number of participants included in the analysis. For dichotomous data we recorded the number of patients rated as responders and the total number of participants included in the study.

Study quality

Two reviewers (F.G. and I.F.) independently assessed the quality of the included studies using the Downs & Black checklist, an instrument suitable for assessing the methodological quality of randomised trials and non-randomised studies. Reference Downs and Black9 The scale is composed of 27 criteria and assesses five domains: study reporting, external validity, internal validity (bias), internal validity (confounding) and power. In the present review, the Downs & Black checklist was used in a modified version in accordance with previous reviews. Reference Samoocha, Bruinvels, Elbers, Anema, van der Beek and Allard10Reference Chudyk, Jutai, Petrella and Speechley12 Specifically, scoring for item 27 dealing with statistical power was simplified to a choice of awarding either one point or zero, depending on whether there was sufficient power to detect a clinically important effect. Reference Samoocha, Bruinvels, Elbers, Anema, van der Beek and Allard10,Reference Chudyk, Jutai, Petrella and Speechley12 As has been done in the other reviews using the Downs & Black scale, Reference Samoocha, Bruinvels, Elbers, Anema, van der Beek and Allard10Reference Chudyk, Jutai, Petrella and Speechley12 we grouped score ranges into the following four levels: excellent (26–28), good (20–25), fair (15–19) and poor (⩽14). The quality level of excellent can be achieved by randomised studies only. We solved disagreements in the quality assessment by discussion.

Data presentation

We graphically represented the effects of the included studies using a modification of the Harvest plot, which is a flexible method suitable for synthesising evidence about differential effects of heterogeneous and complex interventions, across different variables of interest (such as study design and participant characteristics). Reference Ogilvie, Fayter, Petticrew, Sowden, Thomas and Whitehead13,Reference Crowther, Avenell, MacLennan and Mowatt14 In our plots, effect size is represented by a bar, amended by the 95% confidence interval. The study showed a statistically significant effect if the confidence interval is not including the null.

Effect sizes and meta-analyses

Only randomised studies were included in the meta-analysis. We entered and analysed data using comprehensive meta-analysis. We calculated dichotomous outcomes on an intention-to-treat basis, taking into account the initial number of participants included in each study. For the analysis of dichotomous outcomes we used odds ratios (ORs) with 95% confidence intervals, in order to use direct estimates from cluster randomised trials accounting for clustering wherever possible. We analysed continuous data using standardised mean differences (SMDs), as this measure of treatment effect allows combining scores from different scales. If end-point data were unavailable, we analysed change score data. Where intention-to-treat (ITT) data were available we preferred these over ‘per-protocol analysis’. Effect sizes were converted to SMDs for all outcomes in the harvest plot. In contrast to the protocol, we combined effect sizes for provider performance and patient outcomes across diagnoses, because of the small number of studies and the studies with mixed populations.

We investigated statistical heterogeneity by visual inspection of the forest plots. This was supplemented by the I 2 statistic, which provides an estimate of percentage of variability because of heterogeneity rather than chance alone. When the I 2 estimate was greater than or equal to 50%, we interpreted this as indicating the presence of high levels of heterogeneity. Reference Higgins, Julian, Thompson, Deeks and Altman15

Results

Characteristics of included studies

The search yielded a total of 1750 records potentially relevant for this review (online Fig. DS1). Of these, we excluded 1668 records based on the review of titles or abstracts. We retrieved the remaining 82 articles in full text and assessed them for inclusion. Of these, we excluded 63 studies for the reasons reported in online Fig. DS1. The remaining 19 studies Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16Reference Yoshino34 met the inclusion criteria and were included in the present review. The meta-analysis included six RCTs providing data on the impact of guideline implementation on process outcomes Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Biswas and Kilbourne19,Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 and three RCTs on patient outcomes. Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Osborn, Nazareth, Wright and King26 The main study characteristics are presented in online Table DS1.

We included eight RCTs Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Bauer, Biswas and Kilbourne19,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 four CCTs Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32,Reference Yoshino34 and seven studies with a before-and-after design. Reference Baker, Lovell and Harris17,Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Chuang and Crismon21,Reference Mistler, Mellman and Drake24,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 Nine studies enrolled patients with a diagnosis of schizophrenia or related psychotic disorders, Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Chong, Ravichandran, Poon, Soo and Verma20Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Steinacher, Mausolff and Gusy29,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 whereas five studies included participants with unipolar and bipolar affective disorders. Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Bauer, Biswas and Kilbourne19,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32,Reference Yoshino34 One study Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28 included two independent samples of patients diagnosed with schizophrenia or depression. Two studies Reference Mistler, Mellman and Drake24,Reference Osborn, Nazareth, Wright and King26 recruited people with mixed diagnoses (schizophrenia or related psychotic disorders, unipolar and bipolar depression). Diagnosis was not reported in two studies. Reference Baker, Lovell and Harris17,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 The majority of studies Reference Baker, Lovell and Harris17Reference Bauer, Biswas and Kilbourne19,Reference Chuang and Crismon21,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 were conducted in in-patient settings, six studies were performed in out-patient settings, Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32,Reference Yoshino34 one study Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27 reported both in-patient and out-patient settings and two studies Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Osborn, Nazareth, Wright and King26 did not report information on the setting.

Characteristics of the implementation strategies

The implemented guidelines included clinical practice manuals and evidence-based treatment algorithms, which were developed with a structured methodology that included a consensus group of experts, analysis of the evidence base and clinical input. Fifteen studies Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Baker, Lovell and Harris17,Reference Bauer, Biswas and Kilbourne19,Reference Chuang and Crismon21Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 assessed the impact of multifaceted implementation strategies that included two or more components. All studies implemented the guidelines at a professional level, for example by distribution of educational material, educational meetings, educational outreach visits, reminders and audit and feedback. Five studies Reference Bauer, Biswas and Kilbourne19,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32 additionally used strategies at an organisational level targeting patient needs, for example offering an education programme for patients and their families. Two studies Reference Bauer, Biswas and Kilbourne19,Reference Osborn, Nazareth, Wright and King26 used provider-orientated implementation strategies at an organisational level, such as through employment of a nurse as coordinator (online Table DS1). Details about implementation strategies were not available in four studies. Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Mistler, Mellman and Drake24,Reference Yoshino34 Fifteen studies Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25, Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32Reference Yoshino34 compared guideline implementation strategies with usual care, and four randomised studies Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 compared different guideline implementation strategies (online Table DS1). According to the Downs & Black quality check, the quality of included studies was graded as fair in eleven Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Biswas and Kilbourne19,Reference Ince, Tai and Haddock23,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32 and poor in six studies. Reference Baker, Lovell and Harris17,Reference Chong, Ravichandran, Poon, Soo and Verma20Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Yoshino34 Only two studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 achieved a good-quality score.

Impact of guideline implementation on provider performance

Twelve studies assessed the impact of guideline implementation on provider performance Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Baker, Lovell and Harris17,Reference Bauer, Biswas and Kilbourne19Reference Chuang and Crismon21,Reference Mistler, Mellman and Drake24,Reference Osborn, Nazareth, Wright and King26Reference Steinacher, Mausolff and Gusy29,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 (Fig. 1), including four studies comparing implementation strategies. Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 The quality of the majority of studies was judged to be fair.

Fig. 1 Harvest plot.

The studies are grouped according to their quality (poor, fair good) and the two rows indicate whether the outcome refers to patients or provider performance. SMD, standardised mean difference.

Guideline implementation v. treatment as usual

In comparison with usual care, a positive effect of guideline implementation was shown in six studies, Reference Baker, Lovell and Harris17,Reference Bauer, Biswas and Kilbourne19,Reference Chong, Ravichandran, Poon, Soo and Verma20,Reference Mistler, Mellman and Drake24,Reference Steinacher, Mausolff and Gusy29,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 but in only three of them Reference Bauer, Biswas and Kilbourne19,Reference Mistler, Mellman and Drake24,Reference Weinmann, Hoerger, Erath, Kilian, Gaebel and Becker33 was this difference statistically significant. With the exception of one study, Reference Mistler, Mellman and Drake24 showing a large statistically significant effect size, the effect sizes were small to modest. A negative effect of guideline implementation strategies on provider performance was found in four studies. Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Chuang and Crismon21,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29 This negative effect was statistically significant in two studies. Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Chuang and Crismon21 One study Reference Steinacher, Mausolff and Gusy29 examined two different implementation strategies (active v. passive), but did not compare these strategies directly. The study showed a small positive effect size for the active dissemination strategy and, by contrast, a larger negative effect for the passive strategy, but both results were not statistically significant. Reference Steinacher, Mausolff and Gusy29

Two RCTs comparing guideline implementation with usual care were included in the meta-analysis. Reference Baandrup, Allerup, Lublin, Nordentoft, Peacock and Glenthoj16,Reference Bauer, Biswas and Kilbourne19 These studies (Fig. 2) did not show a statistically significant effect of guideline implementation on provider performance (OR = 1.01, 95% CI 0.37–2.79) (Fig. 2). The statistical heterogeneity of these two studies was high (I 2 = 91%).

Fig. 2 Forest plot for process outcomes.

s.e., standard error; IV, inverse variance; CI, confidence interval; TAU, treatment as usual.

Enhanced guideline implementation v. basic implementation strategy

All four studies Reference Ince, Tai and Haddock23,Reference Osborn, Nazareth, Wright and King26,Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 comparing different guideline strategies showed positive effect sizes, but in two of these Reference Owen, Hudson, Thrush, Thapa, Armitage and Landes27,Reference Thompson, Sullivan, Barley, Strange, Moore and Rogers31 the effect was negligible. One study Reference Osborn, Nazareth, Wright and King26 showed a large effect size and a statistical significant advantage of the enhanced implementation strategy (Fig. 2). All four studies were included in the meta-analysis, which did not show a statistically significant advantage of enhanced implementation strategies (OR = 1.47, 95% CI 0.86 to 2.52) (Fig. 2).

Impact of guideline implementation on patient outcomes

Guideline implementation v. treatment as usual

The impact of guideline implementation on patient outcomes was investigated in ten studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22,Reference Mistler, Mellman and Drake24,Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32Reference Yoshino34 (Fig. 1). Three studies Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Suppes, Rush, Dennehy, Crismon, Kashner and Toprac30,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32 presented the results of the same project separately for patients with different diagnoses. The quality of the majority of studies was judged to be fair. Two studies Reference Sorensen, Nielsen, Stage, Brøsen and Damkier28,Reference Steinacher, Mausolff and Gusy29 showed negative effects of guideline implementation on patient outcomes. Four non-randomised studies Reference Miller, Hall, Buchanan, Buckley, Chiles and Conley25,Reference Trivedi, Rush, Crismon, Kashner, Toprac and Carmody32Reference Yoshino34 showed a statistically significant effect in favour of the guideline implementation strategy (Fig. 1). With the exception of one study, Reference Steinacher, Mausolff and Gusy29 showing a large and statistically significant, but negative effect for a passive implementation strategy, effect sizes were small to modest. Two randomised studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18,Reference Hamann, Langer, Winkler, Busch, Cohen and Leucht22 were included in the meta-analysis of patient outcomes (Fig. 3). The meta-analysis did not reveal a statistically significant effect of guideline implementation on patient outcomes (OR = 1.46, 95% CI 0.91–2.35). There was no statistical between-study heterogeneity (I 2 = 0%) (Fig. 3).

Fig. 3 Forest plot for patient outcomes.

s.e., standard error; IV, inverse variance; CI, confidence interval; TAU, treatment as usual.

Enhanced guideline implementation v. basic implementation strategy

One RCT comparing two implementation strategies assessed patient outcomes. Reference Osborn, Nazareth, Wright and King26 The effect reported in this study was not statistically significant (OR = 1.62, 95% CI 0.83–3.15), but numerically in favour of the enhanced guideline strategy (Fig. 3).

Discussion

Main findings

The present systematic review revealed that the pathway from evidence-based guidelines to evidence-based practice is still a neglected research area and that convincing evidence for beneficial effects of guideline implementation is scant. This is consistent with results from previous studies in other fields of medicine, Reference Karbach, Schubert, Hagemeister, Ernstmann, Pfaff and Höpp35Reference Hagemeister, Schneider, Barabas, Schadt, Wassmer and Mager37 which showed that guidelines are still not adequately implemented. Only a minority of the studies included in our review showed a positive statistically significant effect of guideline implementation on provider performance or patient outcomes, and often these studies employed a non-randomised design. In contrast to previous reviews, our data, including the meta-analyses of randomised trials, do not suggest an impact of guideline implementation on provider performance. In other words, efforts to implement guidelines did not modify healthcare provider performance in terms of better adherence to clinical practice guidelines. Despite this finding, the studies showed a more or less consistent positive effect of guideline implementation on patient outcomes, i.e. clinical condition, remission rate and satisfaction with care. Although the meta-analysis did not reveal a statistically significant effect of guideline implementation strategies compared with usual treatment or basic implementation strategies, the magnitude of the effect sizes and the consistency of the effects across studies seem to speak in favour of an effect on patient outcomes, which may not have been revealed as a result of low power.

Interpretation of our findings

The surprising result of a missing effect of guideline implementation on guideline adherence may, in part, be explained by differences between the studies included in meta-analyses and by the methodological difficulties of evaluating complex interventions. One reasonable explanation may be that guideline adherence is already high in treatment-as-usual patient groups, and thus may not be sensitive for change. However, this raises the issue of why a majority of studies have suggested positive effects on patient outcomes. Modest clinical improvement of patients may reflect a Hawthorne effect. However, the development of guidelines based on RCTs is criticised for poor external validity, as a result of highly selective inclusion criteria and selective trial settings. Reference Rothwell38 Furthermore, in a small survey of clinicians in the USA Perlis et al Reference Perlis39 found that the most frequently cited reason for not using guidelines was that they do not adequately reflect features of patients that are relevant in making treatment decisions. Thus, the effect on patient outcomes may reflect (some) thoughtful and well-founded clinical decisions of non-adherence for a particular patient. To shed light on the limited effects of guideline implementation, reasons for guideline deviations should be researched. Only one of the studies Reference Bauer, Pfennig, Linden, Smolka, Neu and Adli18 included in our review reported having documented reasons for non-adherence with the guideline in the intervention group, but these results were not included in the analyses. Clinical practice guidelines neither are intended to overrule practitioners' experience, nor are they unequivocal reflections of the scientific evidence. Our review reveals a clear gap of knowledge on how guidelines may influence the process of clinical care. Nevertheless, if quality of care is understood as actions increasing the likelihood that best patient outcomes are reached, our results challenge the use of guideline adherence as a quality indicator.

In the light of the low level of evidence in general, we were not able to determine which implementation strategies perform best. Both studies that were showing a substantial and significant effect on provider performance implemented the guideline at an organisational level by introducing a nurse as coordinator. Reference Bauer, Biswas and Kilbourne19,Reference Osborn, Nazareth, Wright and King26 However, our results do not clearly indicate that multifaceted implementation strategies are superior to simple strategies. This is consistent with findings from a systematic review of guideline implementation strategies in other fields of medicine that failed to report either superiority of multifaceted interventions over other types of implementation or any relationship of number of intervention components and implementation effect. Reference Grimshaw, Thomas, MacLennan, Fraser, Ramsay and Vale2 Furthermore, evidence on the cost-effectiveness of such organisational changes is currently not available.

Limitations

The present systematic review has some limitations. First, the included studies were substantially heterogeneous in terms of the focus of the guideline, target of the intervention, population, implementation strategies and control group definition. Consequently, it may be difficult to assume that an implementation strategy that proved to be successful in a specific local setting would be similarly successful in a different setting. A second concern is the selection of only one outcome for provider performance and one patient outcome for the purposes of this review (although most studies reported several outcomes). This selection process was based on the background logic of focusing on primary outcomes, but we acknowledge that the included studies analysed a wide range of secondary outcomes that generated clinically interesting insights. The search strategy may have missed some studies as publications may not have used common keywords or may have used keywords that our search failed to capture. Furthermore, the definition of guideline that we employed (systematically developed statements or algorithms, flow charts, tables to assist decisions about appropriate healthcare for specific clinical circumstances) Reference Barbui, Girlanda, Ay, Cipriani, Becker and Koesters4 left some subjectivity in deciding whether a publication or document could be considered a guideline.

Implications

In conclusion, mental healthcare professionals are faced with little safe ground regarding the best use of available guidelines, and further efforts should be made to understand the effects of guidelines on clinical practice. Also, developing guidelines and having them available (and performing studies on their implementation) could change routine practice in domains not related or not coinciding with provider-guideline adherence. If having guidelines available or being involved in studies on guideline implementation were related to improvements in clinician (or team) motivation, qualification or behaviour, patient outcomes might improve without enhancing guideline adherence. This possibility highlights the conundrum of how clinical effects come about and what makes clinicians and interventions effective in everyday practice.

Funding

This research was supported by a grant no. from the German Federal Ministry of Education and Research. We thank Professor Steinacher for providing additional data.

Footnotes

Declaration of interest

None.

References

1 Fretheim, A, Schünemann, HJ, Oxman, AD. Improving the use of research evidence in guideline development: 15. Disseminating and implementing guidelines. Health Res Policy Syst 2006; 4: 27.Google Scholar
2 Grimshaw, JM, Thomas, RE, MacLennan, G, Fraser, C, Ramsay, CR, Vale, L, et al Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004; 8: iiiiv, 1–72.Google Scholar
3 Weinmann, S, Koesters, M, Becker, T. Effects of implementation of psychiatric guidelines on provider performance and patient outcome: systematic review. Acta Psychiatr Scand 2007; 115: 420–33.Google Scholar
4 Barbui, C, Girlanda, F, Ay, E, Cipriani, A, Becker, T, Koesters, M. Implementation of treatment guidelines for specialist mental health care. Cochrane Database Syst Rev 2014; 1: CD009780.Google Scholar
5 Girlanda, F, Fiedler, I, Ay, E, Barbui, C, Koesters, M. Guideline implementation strategies for specialist mental healthcare. Curr Opin Psychiatry 2013; 26: 369–75.Google Scholar
6 Koesters, M, Girlanda, F, Ay, E, Cipriani, A, Barbui, C. Implementation of Treatment Guidelines for Specialist Mental Health Care Of Severely Mentally Ill Patients. Ulm University, 2012 (http://vts.uni-ulm.de/docs/2012/8216/vts_8216_12020.pdf).Google Scholar
7 Liberati, A, Altman, DG, Tetzlaff, J, Mulrow, C, Gøtzsche, PC, Ioannidis, , et al The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 2009; 62: e134.Google Scholar
8 Cochrane Effective Practice and Organisation of Care Group. Data Collection Checklist. EPOC, 2002 (http://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/datacollectionchecklist.pdf).Google Scholar
9 Downs, SH, Black, N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Comm Health 1998; 52: 377–84.Google Scholar
10 Samoocha, D, Bruinvels, DJ, Elbers, NA, Anema, JR, van der Beek, , Allard, J. Effectiveness of web-based interventions on patient empowerment: a systematic review and meta-analysis. J Med Internet Res 2010; 12: e23.Google Scholar
11 Hooper, P, Jutai, JW, Strong, G, Russell-Minda, E. Age-related macular degeneration and low-vision rehabilitation: a systematic review. Can J Ophthalmol 2008; 43: 180–7.Google Scholar
12 Chudyk, AM, Jutai, JW, Petrella, RJ, Speechley, M. Systematic review of hip fracture rehabilitation practices in the elderly. Arch Phys Med Rehabil 2009; 90: 246–62.Google Scholar
13 Ogilvie, D, Fayter, D, Petticrew, M, Sowden, A, Thomas, S, Whitehead, M, et al The harvest plot: a method for synthesising evidence about the differential effects of interventions. BMC Med Res Methodol 2008; 8: 8.Google Scholar
14 Crowther, M, Avenell, A, MacLennan, G, Mowatt, G. A further use for the Harvest plot: a novel method for the presentation of data synthesis. Res Synth Methods 2011; 2: 7983.Google Scholar
15 Higgins, , Julian, PT, Thompson, SG, Deeks, JJ, Altman, DG. Measuring inconsistency in meta-analyses. BMJ 2003; 327: 557–60.Google Scholar
16 Baandrup, L, Allerup, P, Lublin, H, Nordentoft, M, Peacock, L, Glenthoj, B. Evaluation of a multifaceted intervention to limit excessive antipsychotic co-prescribing in schizophrenia out-patients. Acta Psychiatr Scand 2010; 122: 367–74.Google Scholar
17 Baker, JA, Lovell, K, Harris, N. The impact of a good practice manual on professional practice associated with psychotropic PRN in acute mental health wards: an exploratory study. Int J Nurs Stud 2008; 45: 1403–10.Google Scholar
18 Bauer, M, Pfennig, A, Linden, M, Smolka, MN, Neu, P, Adli, M. Efficacy of an algorithm-guided treatment compared with treatment as usual: a randomized, controlled study of inpatients with depression. J Clin Psychopharmacol 2009; 29: 327–33.Google Scholar
19 Bauer, MS, Biswas, K, Kilbourne, AM. Enhancing multiyear guideline concordance for bipolar disorder through collaborative care. Am J Psychiatry 2009; 166: 1244–50.Google Scholar
20 Chong, S, Ravichandran, N, Poon, L, Soo, K, Verma, S. Reducing polypharmacy through the introduction of a treatment algorithm: use of a treatment algorithm on the impact on polypharmacy. Ann Acad Med Singapore 2006; 35: 457–60.Google Scholar
21 Chuang, WC, Crismon, ML. Evaluation of a schizophrenia medication algorithm in a state hospital. Am J Health Syst Pharm 2003; 60: 1459–67.Google Scholar
22 Hamann, J, Langer, B, Winkler, V, Busch, R, Cohen, R, Leucht, S, et al Shared decision making for in-patients with schizophrenia. Acta Psychiatr Scand 2006; 114: 265–73.Google Scholar
23 Ince, P, Tai, S, Haddock, G. Using plain English and behaviourally specific language to increase the implementation of clinical guidelines for psychological treatments in schizophrenia. J Ment Health 2015; 24: 129–33.Google Scholar
24 Mistler, LA, Mellman, TA, Drake, RE. A pilot study testing a medication algorithm to reduce polypharmacy. Qual Saf Health Care 2009; 18: 55–8.Google Scholar
25 Miller, AL, Hall, CS, Buchanan, RW, Buckley, PF, Chiles, JA, Conley, RR, et al The Texas Medication Algorithm Project antipsychotic algorithm for schizophrenia: 2003 update. J Clin Psychiatry 2004; 65: 500–8.Google Scholar
26 Osborn, DPJ, Nazareth, I, Wright, CA, King, MB. Impact of a nurse-led intervention to improve screening for cardiovascular risk factors in people with severe mental illnesses. Phase-two cluster randomised feasibility trial of community mental health teams. BMC Health Serv Res 2010; 10: 61.Google Scholar
27 Owen, RR, Hudson, T, Thrush, C, Thapa, P, Armitage, T, Landes, RD. The effectiveness of guideline implementation strategies on improving antipsychotic medication management for schizophrenia. Med Care 2008; 46: 686–91.Google Scholar
28 Sorensen, L, Nielsen, B, Stage, KB, Brøsen, K, Damkier, P. Implementation of a rational pharmacotherapy intervention for inpatients at a psychiatric department. Nordic J Psychiatry 2008; 62: 242–9.Google Scholar
29 Steinacher, B, Mausolff, L, Gusy, B. The effects of a clinical care pathway for schizophrenia: a before and after study in 114 patients. Dtsch Arztebl Int 2012; 109: 788–94.Google Scholar
30 Suppes, T, Rush, AJ, Dennehy, EB, Crismon, ML, Kashner, TM, Toprac, MG, et al Texas Medication Algorithm Project, phase 3 (TMAP-3): clinical results for patients with a history of mania. J Clin Psychiatry 2003; 64: 370–82.Google Scholar
31 Thompson, A, Sullivan, SA, Barley, M, Strange, SO, Moore, L, Rogers, P, et al The DEBIT trial: an intervention to reduce antipsychotic polypharmacy prescribing in adult psychiatry wards – a cluster randomized controlled trial. Psychol Med 2008; 38: 705–15.Google Scholar
32 Trivedi, MH, Rush, AJ, Crismon, ML, Kashner, TM, Toprac, MG, Carmody, TJ, et al Clinical results for patients with major depressive disorder in the Texas Medication Algorithm Project. Arch Gen Psychiatry 2004; 61: 669–80.Google Scholar
33 Weinmann, S, Hoerger, S, Erath, M, Kilian, R, Gaebel, W, Becker, T. Implementation of a schizophrenia practice guideline: clinical results. J Clin Psychiatry 2008; 69: 1299–306.Google Scholar
34 Yoshino, A. Algorithm-guided treatment versus treatment as usual for major depression. Psychiatry Clin Neurosci 2009; 5: 652–7.Google Scholar
35 Karbach, U, Schubert, I, Hagemeister, J, Ernstmann, N, Pfaff, H, Höpp, H. Physicians' knowledge of and compliance with guidelines: an exploratory study in cardiovascular diseases. Dtsch Arztebl Int 2011; 108: 61–9.Google Scholar
36 Hagemeister, J, Schneider, CA, Diedrichs, H, Mebus, D, Pfaff, H, Wassmer, G, et al Inefficacy of different strategies to improve guideline awareness – 5-year follow-up of the hypertension evaluation project (HEP). Trials 2008; 9: 39.Google Scholar
37 Hagemeister, J, Schneider, CA, Barabas, S, Schadt, R, Wassmer, G, Mager, G, et al Hypertension guidelines and their limitations – the impact of physicians' compliance as evaluated by guideline awareness. J Hypertension 2001; 19: 2079–86.Google Scholar
38 Rothwell, PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet 2005; 365: 8293.Google Scholar
39 Perlis, RH. Use of treatment guidelines in clinical decision making in bipolar disorder: a pilot survey of clinicians. Curr Med Res Opin 2007; 23: 467–75.Google Scholar
Figure 0

Fig. 1 Harvest plot.The studies are grouped according to their quality (poor, fair good) and the two rows indicate whether the outcome refers to patients or provider performance. SMD, standardised mean difference.

Figure 1

Fig. 2 Forest plot for process outcomes.s.e., standard error; IV, inverse variance; CI, confidence interval; TAU, treatment as usual.

Figure 2

Fig. 3 Forest plot for patient outcomes.s.e., standard error; IV, inverse variance; CI, confidence interval; TAU, treatment as usual.

Supplementary material: PDF

Girlanda et al. supplementary material

Supplementary Material

Download Girlanda et al. supplementary material(PDF)
PDF 527.8 KB
Supplementary material: PDF

Girlanda et al. supplementary material

Supplementary Table S1

Download Girlanda et al. supplementary material(PDF)
PDF 824.7 KB
Submit a response

eLetters

No eLetters have been published for this article.