Hostname: page-component-7c8c6479df-xxrs7 Total loading time: 0 Render date: 2024-03-18T11:27:57.679Z Has data issue: false hasContentIssue false

Effect of feedback of treatment outcome in specialist mental healthcare: meta-analysis

Published online by Cambridge University Press:  02 January 2018

Carina Knaup
Affiliation:
Department of Psychiatry II, Ulm University, Ulm, Germany
Markus Koesters
Affiliation:
Department of Psychiatry II, Ulm University, Ulm, Germany
Dorothea Schoefer
Affiliation:
Department of Psychiatry II, Ulm University, Ulm, Germany
Thomas Becker
Affiliation:
Department of Psychiatry II, Ulm University, Ulm, Germany
Bernd Puschner*
Affiliation:
Department of Psychiatry II, Ulm University, Ulm, Germany
*
Bernd Puschner, Ulm University, Department of Psychiatry II, Ludwig-Heilmeyer-Str. 2, 89312 Günzburg, Germany. Email: bernd.puschner@bkh-guenzburg.de
Rights & Permissions [Opens in a new window]

Abstract

Background

Feedback of treatment outcome during the course of therapy (outcome management) is increasingly considered to be beneficial for improving the quality of mental healthcare.

Aims

To review the impact of feedback of outcome to practitioners and/or patients in specialist mental health services.

Method

A systematic search and meta-analysis of controlled trials using outcome management in mental health services published in English or German language.

Results

Twelve studies met inclusion criteria. Feeding back outcome showed a small, but significant (d = 0.10; 95% CI 0.01–0.19) positive short-term effect on the mental health of individuals that did not prevail in the long run. Subgroup analysis revealed no significant differences regarding feedback modalities. Outcome management did not contribute to a reduction of treatment duration.

Conclusions

Evidence on the effects of outcome management in mental healthcare is promising. More targeted research is needed in order to identify the effective ingredients of outcome feedback and to assess its cost-effectiveness.

Type
Review Articles
Copyright
Copyright © Royal College of Psychiatrists, 2009 

In many countries, predominantly in the USA, Australia and to a growing degree in Western Europe, the complex management of mental healthcare requires quality assurance and ongoing treatment monitoring. Although routine data collection in mental healthcare is becoming more widespread, these data are rarely fed back to practitioners – let alone to patients – in a meaningful way.

Continuous monitoring of patient outcome was recommended as a means to improve quality of health services 20 years ago. Reference Ellwood1 Ten years later, Lyons et al Reference Lyons, Howard, O'Mahoney and Lish2 presented a concept on how to combine measurement with management of clinical outcomes in mental healthcare. This approach has been refined in the meantime, for example for quality management in psychotherapy. Reference Kordy and Bauer3 Feedback of patient outcome to stakeholders constitutes the prime instrument of outcome management and can be regarded as the most direct link between research and practice since outcome data are being used immediately in order to improve service provision.

Two reviews have summarised the evidence on feedback of outcome in general healthcare Reference Espallargues, Valderas and Alonso4,Reference Valderas, Kotzeva, Espallargues, Guyatt, Ferrans and Halyard5 and found some positive effects on the process of care, but none on patient health status. By focusing on specific settings, Reference Gilbody, House and Sheldon6 interventions Reference Lambert, Whipple, Hawkins, Vermeersch, Nielsen and Smart7 or diagnostic groups, Reference Gilbody, House and Sheldon8 the scope of reviews on the impact of outcome management in mental healthcare has been rather restricted. Thus, evidence on the effectiveness of feedback in mental healthcare is inconclusive: although Gilbody et al Reference Gilbody, House and Sheldon8 could not find any randomised controlled trials (RCTs) investigating the effect of use of outcome measures in the routine treatment of people with schizophrenia, Lambert et al Reference Lambert, Whipple, Hawkins, Vermeersch, Nielsen and Smart7 stated that in out-patient psychotherapy ‘it may be time for clinicians to routinely track and formally monitor patient treatment response’. Furthermore, in many reviews, information from outcome measures for screening purposes has been mixed with continuous outcome management. Thus, there is a need for a systematic review of the effectiveness of feedback of outcome in specialist mental healthcare.

Method

Search strategy

A systematic literature search was conducted in the electronic databases Medline, PSYNDEX, PsycINFO, the Cochrane Central Register of Controlled Trials, the Cochrane Database of Systematic Reviews, the Current Controlled Trials Register, and the world wide web using Google and Google Scholar in November 2006. This search was updated in March 2008 using the same search criteria. A broad set of search terms was applied in order to achieve maximum sensitivity. Keywords entered were ‘mental disorders’ combined with ‘feedback’ and ‘treatment outcome’ as well as diverse synonyms of these search terms. A total of 1066 publications were identified in this first step. Additional hand searching was carried out using the reference lists of relevant articles and appropriate reviews. Reference Espallargues, Valderas and Alonso4,Reference Gilbody, House and Sheldon6,Reference Marshall, Haywood and Fitzpatrick9,Reference Sapyta, Riemer and Bickman10 If data were missing or not apt for use in meta-analysis, authors were contacted for further information (details of the search strategy and the search terms used can be obtained from B.P.).

Inclusion criteria

English- as well as German-language publications were included in this systematic review if they examined feedback of outcome in specialist mental healthcare settings. Study participants had to be of adult age with mental health problems treated in psychiatric or psychotherapeutic settings (community, in-patient, and out-patient mental health services). Feedback was defined as providing mental healthcare professionals and/or patients with individual information on treatment outcome based on standardised measures (e.g. mental health, unmet needs). Only studies with a controlled design (not necessarily randomised) that evaluated effects of feedback interventions on patient outcome such as mental health were selected for analysis. Two of the authors (C.K. and B.P.) independently decided about study inclusion. Disagreements were resolved by discussion.

Exclusion criteria

Duplicate publications, abstracts and studies in which outcome assessment was used exclusively for routine screening or diagnosis were excluded.

Data extraction

Study characteristics, features of the feedback systems and outcome data were extracted from each included article (by C.K. and subsequently checked by B.P.) using a standard coding form developed for this review. This form included authors, publication year, study type, country, setting, patient and therapist population, unit of randomisation, inclusion and exclusion criteria, groups, feedback and effect instruments, measurement points, feedback intervention (duration, type and content of feedback, recipient, frequency and timing), outcomes assessed, as well as results and method of analysis. Furthermore, cost data (e.g. treatment duration) were noted if available.

Procedures

Data synthesis was conducted using the following two-step procedure: first, a qualitative analysis was conducted in order to provide a narrative overview. Second, a quantitative data synthesis (meta-analysis) was carried out separately for all trials with sufficient data on mental health outcome and treatment duration as end-points of the intervention. All symptoms scales applied in the studies were used as a measure of mental health outcome. If studies reported more than one scale or compound dimensions of an outcome, a single mean effect size was calculated for each study in order to avoid bias in favour of trials with multiple outcome measures.

Given our focus on differences in post-treatment outcome in controlled studies, effect sizes were calculated on the basis of (raw-) post-score analysis using the formula d =(M 1M 2)/s.d., where M 1 and M 2 are the means of the outcome measures in the intervention (having received feedback) and control groups, and s.d. is the pooled standard deviation of the two groups' post-scores. These effect sizes were corrected in order to obtain unbiased estimators using procedures suggested by Hedges. Reference Hedges11 The direction of effect was standardised so that a positive value indicates a better outcome in the feedback condition. As larger sample sizes lead to more reliable estimators, all calculated effect sizes were weighted by the inverse of their variance. Reference Hedges and Oklin12

Some studies reported the results of more than two groups. Reference Lambert, Whipple, Vermeersch, Smart, Hawkins and Nielsen13Reference Lambert, Whipple, Smart, Vermeersch, Nielsen and Hawkins15 In these cases, effect sizes of each control group comparison were pooled, resulting in one effect size per trial. An exception was made in a study with subsequent phases Reference Bauer16 – each with an intervention and a control group – where both study phases were treated as two independent trials and effect sizes were calculated separately.

Heterogeneity between and within the selected studies was anticipated as a result of variation in settings, feedback interventions, outcome instruments, the mental disorders the participants had and mental health professionals' therapeutic orientation. Therefore, the meta-analysis was performed using a random-effects model as suggested by DeSimonian & Laird Reference DerSimonian and Laird17 that considers both intra- and inter-study variability. To quantify heterogeneity, the I 2 statistic, which represents an estimation of the percentage of total variation across studies due to heterogeneity, was calculated. I 2 values below 25% indicate low heterogeneity whereas those above 75% point towards high heterogeneity. Reference Higgins, Thompson, Deeks and Altmann18

To prevent dilution of effects as a result of differences in length of follow-up periods within one trial or in comparison with the others, we calculated two separate meta-analyses: short-term for follow-up intervals up to 9 weeks after intake (predominantly at the end of therapy), and long-term for those between 3 and 12 months.

Evidence of publication bias was analysed using graphical control techniques (funnel plots) and Egger's regression test. Reference Egger, Davey Smith, Schneider and Minder19 A fail-safe number was calculated in view of the fact that non-significant studies tend to be unpublished (‘file-drawer problem’). This parameter estimates the number of studies with an effect size of zero required to nullify the overall effect size. To test the robustness of the overall effect and to prove the influence of single trials on the overall effect an exclusion sensitivity plot was calculated.

The influence of variables such as the perspective of outcome rating (self v. staff), setting and feedback modalities, including recipient, content, frequency and timing of feedback, on the effects of feedback interventions was explored via a priori defined subgroup analyses. All analyses were performed with Comprehensive Meta Analysis 2.0 for Windows and the overall effects are presented as Hedges' g with 95% confidence intervals (CI). Reference Hedges and Oklin12

Results

Search flow

The search strategy initially identified 1078 publications. After initial title and abstract screening, 68 studies were rated as potentially relevant and screened in full text. Short-listed trials that were excluded were mostly descriptive reports dealing with outcome monitoring or with theoretical aspects, development, implementation, feasibility or acceptance of feedback systems. Of the remaining trials, only 21 fulfilled the inclusion criteria. Three of them could not be retrieved even after efforts to contact the authors, one was written in a language other than English or German, and five were still in progress with no published outcome data reported.

Characteristics of included studies

Finally, 12 studies published between 2001 and 2006 met all selection criteria and were available for meta-analysis (Tables 1 and 2). Half of them were conducted in the USA, four in the UK, and two in Germany. Two studies were dissertations. Reference Bauer16,Reference Trudeau20 Sample size varied between 127 and 1374 participants. Most studies were conducted in out-patient settings such as community-based mental health services (five studies), university counselling centres (three studies), and specialist mental health units (old age psychiatry, eating disorder; two studies). Only two trials were performed in in-patient psychotherapeutic clinics. Reference Bauer16,Reference Berking, Orth and Lutz21

Table 1 Study properties

Properties n %
Location
    USA 6 50
    UK 4 33
    Germany 2 17
Setting
    In-patient 2 17
    Out-patient 10 83
Design
    Randomised controlled trial 10 83
    Controlled trial 2 17
Patients
    Mental illness (mixed diagnoses) 10 83
    Old age 1 8
    Eating disorders 1 8
Unit of randomisation
    Patients 7 58
    Professionals 1 8
    Patients and professionals 2 17
    None 2 17

Table 2 Characteristics of the included studies and their participants

Country and setting Design; intervention group/control group n; follow-up time points n Age, years (mean) Female, %a Illness Outcome measures Feedback to
Ashaye et al (2003) Reference Ashaye, Livingston and Orrell26 UK day hospital RCT; 1/1; 3 months 112 ≥65 (76.4) 64 Depression, dementia HoNOS (65+), CAPE—BRS Staff
Bauer (2004) Reference Bauer16 Germany in-patient CT; 2/2; discharge 391 18-79 (34.9) 71 Personality disorder, depression, anxiety SCL—11, OQ—45, IS Clinician
Berking et al (2006) Reference Berking, Orth and Lutz21 Germany in-patient RCT; 1/1; discharge 118 Adults (49.4) 62 Depression, anxiety FEP, VEV, CGI Clinician
Brodey et al (2005) Reference Brodey, Cuffel, McCulloch, Tani, Maruish and Brodey23 USA out-patient RCT; 1/1; 6 weeks 1374 ≥18 75 Depression, anxiety SCL—11 Clinician
Hawkins et al (2004) Reference Hawkins, Lambert, Vermeersch, Slade and Tuttle14 USA out-patient RCT; 2/1; discharge 201 Adults (30.8) 68 Mood and anxiety disorders OQ—45 Clinician or clinician and patient
Lambert et al (2001) Reference Lambert, Whipple, Smart, Vermeersch, Nielsen and Hawkins15 USA out-patient RCT; 1/1; discharge 609 17-57 (22.2) 70 Personal concerns OQ—45 Clinician
Lambert et al (2002) Reference Lambert, Whipple, Vermeersch, Smart, Hawkins and Nielsen13 USA out-patient CT; 1/1; discharge 1020 17-57 (22.3) 70 Personal concerns OQ—45 Clinician
Marshall et al (2004) Reference Marshall, Lockwood, Green, Zajac-Roles, Roberts and Harrison27 UK out-patient RCT; 2/1; 12 months 304 Adults Schizophrenia, depression BPRS, WHO—DAS Care coordinator
Schmidt et al (2006) Reference Schmidt, Landau, Pombo-Carril, Bara-Carril, Reid and Murray22 UK out-patient RCT; 1/1; discharge and 6 months 61 Adults 100 Eating disorder SEED Patient
Slade et al (2006) Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi24 UK out-patient RCT; 1/1; 7 months 160 18-64 (41.2) 22 Schizophrenia, affective disorder BPRS, HoNOS, TAG Staff and patient
Trudeau (2001) Reference Trudeau20 USA out-patient RCT; 1/2; 2 and 4 months 127 Adults (33.9) 72 Mental health problems OQ—45, RAND—36 Clinician
Whipple et al (2003) Reference Whipple, Lambert, Vermeersch, Smart, Nielsen and Hawkins25 USA out-patient RCT; 1/1; discharge 981 18-54 (22.9) 66 Personal concerns OQ—45 Clinician

The majority of the studies (10, 83%) were RCTs, in 7 of which the unit of randomisation was the patient. The two non-randomised trials used a controlled prospective cohort design Reference Lambert, Whipple, Vermeersch, Smart, Hawkins and Nielsen13 and a prospective one with sequential study phases. Reference Bauer16 Most of the studies included chose treatment as usual (TAU) for the control groups. One study compared feedback with a control condition that consisted of additional mental health assessment. Reference Schmidt, Landau, Pombo-Carril, Bara-Carril, Reid and Murray22 In thecaseof another study that reported results of two control conditions, data abstraction was restricted to the assessment-only procedure. Reference Trudeau20

Procedures and features of the feedback systems

Characteristics of the feedback systems used are summarised in Table 3. In most studies, feedback was given continuously at 1- or 2-weekly intervals during the course of therapy. Others limited the number of feedback interventions to a maximum of two. Reference Brodey, Cuffel, McCulloch, Tani, Maruish and Brodey23,Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi24 In general, standardised assessments of psychological functioning (e.g. Outcome Questionnaire–45.2, Symptom Checklist–11) or needs (e.g. Camberwell Assessment of Need for the Elderly, Camberwell Assessment of Need Short Appraisal Schedule, Cardinal Needs Schedule) were completed by participants and results were fed back to therapists or staff. Only three studies investigated the provision of a (identical or different) message to patients. Reference Hawkins, Lambert, Vermeersch, Slade and Tuttle14,Reference Schmidt, Landau, Pombo-Carril, Bara-Carril, Reid and Murray22,Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi24

Table 3 Feedback systems useda

Measures used for feedback Modalities of feedback
Rated by Timingb Frequency Content Features
Ashaye et al (2003) Reference Ashaye, Livingston and Orrell26 CANE Clinician Once Unmet needs and suitable interventions List
Bauer (2004) Reference Bauer16 OQ—45, GBB, CSC, PAE, HAQ, FLZ Patient Timely Once Phase 1: Early treatment response; Graphs, tables, textc,d
HAQ, ICD—10 Clinician Phase 2: Additionally prognosis of treatment duration, treatment recommendation
Berking et al (2006) Reference Berking, Orth and Lutz21 FEP Patient Timely Weekly Progress, goal achievement Progress graph, percentage of goal achievement
Brodey et al (2005) Reference Brodey, Cuffel, McCulloch, Tani, Maruish and Brodey23 SCL—11 Patient Timely Twice Progress, extreme answers, fill-in time Graph, tables, textc
Hawkins et al (2004) Reference Hawkins, Lambert, Vermeersch, Slade and Tuttle14 OQ—45 Patient Delayed Weekly Progress, treatment recommendation Graph, coloured progress markers, textc
Lambert et al (2001) Reference Lambert, Whipple, Smart, Vermeersch, Nielsen and Hawkins15 OQ—45 Patient Delayed Weekly Progress, treatment recommendation Graph, coloured progress markers, textc
Lambert et al (2002) Reference Lambert, Whipple, Vermeersch, Smart, Hawkins and Nielsen13 OQ—45 Patient Delayed Weekly Progress, treatment recommendation Graph, coloured progress markers, textc
Marshall et al (2004) Reference Marshall, Lockwood, Green, Zajac-Roles, Roberts and Harrison27 CNS Patient Delayed Once Needs, required interventions, access to intervention Text
Schmidt et al (2006) Reference Schmidt, Landau, Pombo-Carril, Bara-Carril, Reid and Murray22 TREAT—EAT, SEED, HADS Patient Biweekly Physical and psychological status, variables facilitating or hindering change
Slade et al (2006) Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi24 CANSAS, HAS, MANSA CANSAS, HAS, TAG Patient Staff Delayed Twice Progress, areas of disagreement Graphs, text
Trudeau (2001) Reference Trudeau20 OQ—45 Patient Delayed Weekly Progress Graphse
Whipple et al (2003) Reference Whipple, Lambert, Vermeersch, Smart, Nielsen and Hawkins25 OQ—45 Patient Delayed Weekly Progress and treatment response; Graph, coloured progress markers, textc,f
CST for non-responders
CST based upon Haq—II, SCS, MSPSS Patient Delayed Treatment guidelines, CST results Text

Feedback mainly comprised information about current treatment status and changes over time. Thus, the basic component to illustrate treatment progress was graphical and/or numerical data, often complemented with further explanations or in some cases even with treatment recommendations. Only two studies Reference Trudeau20,Reference Berking, Orth and Lutz21 did not use any normative comparison with a reference group to evaluate individual progress in therapy.

Effect variables

A variety of outcomes were evaluated: mental health, met and unmet needs, physical impairment, social functioning, quality of life, patient satisfaction, acceptance or appraisal of feedback, as well as rates of significant clinical change and of treatment response.

Nine of the studies measured mental health status by using patient-reported generic symptom scales. Professional-rated measures of psychological functioning were used in five trials whereas three of them included both raters' perspectives. In half of the studies two or more mental health outcome questionnaires were administered (Table 2).

The parameter most frequently used in order to ascertain savings of direct costs as a result of the use of feedback was treatment duration in terms of number of treatment sessions Reference Lambert, Whipple, Vermeersch, Smart, Hawkins and Nielsen13Reference Lambert, Whipple, Smart, Vermeersch, Nielsen and Hawkins15,Reference Whipple, Lambert, Vermeersch, Smart, Nielsen and Hawkins25 or length of stay. Reference Bauer16 Only one study relied on other indicators of service use (hospital admissions and drop-out rates). Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi24

Short- and long-term effects on mental health outcomes

Most of the studies included measured the impact of feedback on individuals' mental health status at the end of treatment or within 9 weeks after intake. Two of these nine trials also evaluated long-term effects. Reference Trudeau20,Reference Schmidt, Landau, Pombo-Carril, Bara-Carril, Reid and Murray22 The remaining three studies Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi24,Reference Ashaye, Livingston and Orrell26,Reference Marshall, Lockwood, Green, Zajac-Roles, Roberts and Harrison27 reported follow-up measurements only (Table 2).

In the meta-analysis of short-term outcomes, ten group comparisons with a total of 4009 participants could be included. Figure 1 shows the estimated mean effect sizes. The overall effect was d = 0.10 (95% CI 0.01–0.19) favouring the feedback intervention. Heterogeneity across studies was moderate (I 2 = 31%, P = 0.16). There was no substantial change in results (d = 0.11, 95% CI 0.02–0.20, P = 0.02) when limiting this analysis to those trials Reference Lambert, Whipple, Vermeersch, Smart, Hawkins and Nielsen13Reference Bauer16,Reference Trudeau20,Reference Whipple, Lambert, Vermeersch, Smart, Nielsen and Hawkins25 with the outcome measure used most frequently (Outcome Questionnaire–45.2). Reference Lambert, Burlingame, Umphress, Hansen, Vermeersch and Clouse28

Fig. 1 Forest plot for short-term mental health outcome (random-effects model).a a. Treatment outcomes were coded as short-term if they were measured within 9 weeks after initial assessment.

For the examination of long-term effects, five trials with a total of 573 participants were included in the analysis. All except one study Reference Trudeau20 used diverse staff-rated outcome measures that were combined per trial for effect size calculations. As shown in Fig. 2 the mean weighted effect size (d = −0.06, 95% CI 0.22–0.11) was against the expected direction, but it was small and not significant. Heterogeneity between studies was very low (I 2 = 0%, P = 0.69).

Fig. 2 Forest plot for long-term mental health outcome (random-effects model).a a. Treatment outcomes were coded as long-term if they were measured between 3 and 12 months after initial assessment.

Effects on treatment duration

Data on length of treatment as an indicator of costs were available for 981 participants from five studies (42%). Figure 3 shows the results of the pooled meta-analysis. The overall effect size was d = 0.05 (95% CI −0.05 to 0.15) with moderate heterogeneity (I 2 = 42.03%, P = 0.12).

Fig. 3 Forest plot for treatment duration (random-effects model).

Moderator analyses

There were no statistically significant differences in effect sizes between any of the subgroups analysed (Table 4). However, magnitude of subgroup differences was substantial in a number of instances. Feedback was more effective if it: was given to both patients and therapists or staff (v. only to therapists/staff); was reported at least twice (v. only once); comprised information on patient progress (v. on status only). Moreover, only the latter two subgroups showed effect sizes significantly different from zero within their subgroup analysis.

Table 4 Results of subgroup analyses (random-effects model)

Short-term mental health outcome
Subgroup k a Effect size 95% CI P b
Setting
    In-patient 3 0.07 -0.23 to 0.37 0.65
    Out-patient 7 0.11 0.04 to 0.18 0.00
Rater of outcome
    Patient 11 0.11 0.03 to 0.19 0.01
    Staff 3 0.01 -0.42 to 0.44 0.97
Feedback recipient
    Therapist/staff and patient 2 0.30 -0.01 to 0.60 0.06
    Therapist/staff 9 0.09 -0.01 to 0.18 0.07
Feedback content
    Status information 3 -0.04 -0.23 to 0.14 0.63
    Progress information 7 0.13 0.04 to 0.22 0.01
Feedback timingc
    Timely 4 0.06 -0.04 to 0.16 0.23
    Delayed 5 0.06 -0.05 to 0.17 0.31
Feedback frequency
    Once 2 -0.08 -0.27 to 0.11 0.42
    More than once 8 0.13 0.04 to 0.22 0.00

Heterogeneity within the clusters varied between very low and high particularly depending on number of included studies or group comparisons (I 2 ranged from 0 to 84%).

Publication bias and sensitivity analysis

Funnel plots for studies of short- (Fig. 4) or long-term mental health outcome as well as for studies evaluating treatment duration did not provide evidence of publication bias. Fail-safe numbers for the analyses pertaining to short-term effects showed that it would require 11 unpublished null result trials to nullify the calculated effect. Moreover, Egger's regression test was not significant (P = 0.76), suggesting no evidence of small-study bias.

Fig. 4 Funnel plot of estimated effect sizes (Hedges' g) against standard error of the trials for short-term mental health outcome.

Further sensitivity analyses revealed that the exclusion of any single study only marginally changed the overall effect sizes of all three analyses. The greatest difference in short-term outcome analysis occurred when the study of Lambert et al Reference Lambert, Whipple, Vermeersch, Smart, Hawkins and Nielsen13 was removed (d = 0.06, 95% CI −0.02 to 0.15, P = 0.14).

Discussion

Evidence base

Twelve studies fulfilling inclusion criteria for a systematic meta-analysis of the effectiveness of outcome management in mental healthcare were identified. Most of these were conducted in out-patient treatment settings in the USA or the UK. Outcome management in in-patient treatment has only been examined in Germany, which also reflects national differences in mental health service provision. Reference Becker and Vazquez-Barquero29 Sample size averaged 378 participants per study. Altogether, evidence is based upon 4540 adults, two-thirds of them female. The vast majority of study participants were people with affective disorders. However, participants' mental health problems did not always meet standard diagnostic criteria of a mental disorder, especially among student populations on which most of the research of the Lambert group is based. On the other hand, some studies focused on people with severe mental illness (e.g. Slade et al). Reference Slade, McCrone, Kuipers, Leese, Cahill and Parabiaghi24 Given that two decades have passed since the introduction of the general principles of outcome management, Reference Ellwood1 this suggests that some efforts have been made to provide systematic evidence of its effectiveness in specialist mental healthcare.

Short- and long-term effect

Results of this review of controlled studies show that feedback of outcome had a small, albeit statistically significant short-term effect on improving mental health outcomes (d = 0.10, 95% CI 0.01–0.19). This effect was found to be consistent across a variety of outcome measures as analyses limited to studies using a popular single instrument (Outcome Questionnaire–45.2) or excluding the one study with a disorder-specific scale Reference Schmidt, Landau, Pombo-Carril, Bara-Carril, Reid and Murray22 showed similar results. Furthermore, sensitivity analyses revealed that the exclusion of any single study only marginally changed the overall effect sizes of all three analyses, implying relative stability of the results. This corresponds with the calculation of fail-safe numbers indicating that at least 11 null effect trials are needed to diminish the effect. This effect size is similar to that found in other reviews on the topic, for example by Sapyta et al Reference Sapyta, Riemer and Bickman10 on various forms of feedback of client health status information (including screening information only) to health professionals (including general practitioners) in community settings (d = 0.21) or by Lambert et al Reference Lambert, Whipple, Hawkins, Vermeersch, Nielsen and Smart7 on continuous feedback of outcome in out-patient psychotherapy (d = 0.09). Bearing in mind differences in sampling and methods, such comparisons should be viewed cautiously.

It was also found that this effect did not prevail in the long run, i.e. after termination of treatment and outcome feedback, symptomatic impairment in study participants allocated to control groups was no worse than in those who had received outcome management. On the one hand, this finding suggests that outcome management has no persistent effect on improving mental health. On the other, it could also be interpreted to suggest that it might be wise to continue using outcome feedback at least to some extent after the end of treatment in order to avoid wearing off of short-term benefits. Unfortunately, a clear interpretation of this finding is difficult since long-term effects have only rarely been studied (in only five trials).

Furthermore, no advantage of outcome management was found with respect to reduction of treatment costs or cost-offset. This finding should also be interpreted with caution since only very few researchers made efforts to collect comprehensive cost data. Therefore, statistical pooling was limited to studies reporting indicators of the amount of treatment utilised.

Moderators of effect

In addition, apart from specifying the general benefit of outcome management, it is also important to identify its active ingredients. Moderator analyses showed that the effect of outcome management on individuals' short-term mental health outcome is improved if: feedback comprises information on mental health progress (v. only on status); both patient and therapist receive feedback (v. only one of them); and feedback is given frequently (v. only once). Considering the small numbers of studies and substantial heterogeneity for some clusters (I2 ranging from 0 to 84%), these results offer valuable clues but should be interpreted with caution.

Limitations

Even though the aggregated sample size of included studies would appear sufficient, the number of studies these data were drawn from was small and not particularly representative. Additionally, the number of researchers conducting this research is very small, and could introduce bias through a number of avenues. This meta-analysis has a number of further drawbacks. First, assessed studies varied considerably with regard to certain patient characteristics, most notably illness severity and comorbid disorders. The majority of trials relied on data from people with rather mild mental illness treated with out-patient psychotherapy. Second, the number of studies decreased rapidly when we tried to answer more specific questions (e.g. long-term effects, moderators). As a result of variations in study designs, measurement points had to be pooled in order to be able to examine persistence of effect. Third, except in one study, feedback was based on patient-reported outcomes whereas it might be worthwhile to also have data obtained from independent raters. Fourth, in the vast majority of the trials, the unit of randomisation was the patient, which might entail cross-contamination, Reference Gilbody, House and Sheldon8 i.e. ‘real’ effects of the intervention are watered down because of a general sensitisation since one clinician receives feedback for some patients but not for others.

Implications and outlook

Psychotherapy research has put forth a number of rather refined approaches to systematically feeding back patient outcome data to clinicians and also to patients in order to improve quality of mental healthcare, e.g. patient-focused research, Reference Lambert, Hansen and Finch30 expected treatment response Reference Lueger, Howard, Martinovich, Lutz, Anderson and Grissom31 or computer-assisted quality management. Reference Kordy, Hannöver and Richard32 All these outcome management systems heavily rely on professionals' endorsement and use of standardised outcome measures. Although there is a trend towards increased use of such instruments among psychologists in clinical practice, Reference Hatfield and Ogles33 psychiatrists have hardly picked up on this development. Reference Gilbody, House and Sheldon34 A reason for this might be doubts about the benefits of outcome management among clinicians.

This review might contribute to reducing such doubts at least to some extent, since it has shown that feedback of outcome improves symptomatic impairment in people with mental illness treated in various specialist mental health services. Even though overall effect size was modest, some single studies yielded impressive effects. However, the evidence base is hardly sufficient to adequately analyse crucial more specific questions such as differential effectiveness in certain subgroups (e.g. by diagnosis). Nevertheless, this review provides some clues on issues that should be taken into account when implementing outcome management in routine practice, i.e. to feedback outcome continuously and regularly to both clinicians and patients, and to provide information on treatment progress.

Future research might pick up on these clues in order to move beyond trying to answer the question of a general effect of outcome management in mental healthcare. One direction should be to conduct much-needed comprehensive cost-effectiveness analyses based upon sound estimates of costs of feedback (staff, equipment, space and other resource costs), direct and indirect costs and treatment outcome. Most importantly, active ingredients of outcome management should be scrutinised in more detail in order to arrive at feedback systems tailored to the needs of specific subgroups of people with mental illness, but also to the requirements of clinicians who, at least during kick-off, are asked to devote substantial amounts of extra effort to make outcome management work.

Acknowledgements

We thank two anonymous reviewers for their knowledgeable and helpful comments.

Footnotes

This review was carried out during the course of the study ‘Outcome monitoring and outcome management in inpatient psychiatric care’ (EMM), which received funding from the German Federal Ministry of Education and Research (grant number 01GL0504).

Declaration of interest

None.

References

1 Ellwood, PM. Shattuck Lecture – Outcomes management: a technology of patient experience. N Engl J Med 1988; 318: 1549–56.Google ScholarPubMed
2 Lyons, JS, Howard, KI, O'Mahoney, MT, Lish, JD. The Measurement and Management of Clinical Outcomes in Mental Health. Wiley, 1997.Google Scholar
3 Kordy, H, Bauer, S. The Stuttgart-Heidelberg model of active feedback driven quality management: means for the optimization of psychotherapy provision. Int J Clin Health Psych 2003; 3: 615–31.Google Scholar
4 Espallargues, M, Valderas, JM, Alonso, J. Provision of feedback on perceived health status to health care professionals: a systematic review of its impact. Med Care 2000; 38: 175–86.CrossRefGoogle ScholarPubMed
5 Valderas, JM, Kotzeva, A, Espallargues, M, Guyatt, G, Ferrans, CE, Halyard, MY, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res 2008; 17: 179–93.Google ScholarPubMed
6 Gilbody, SM, House, AO, Sheldon, TA. Routinely administered questionnaires for depression and anxiety: systematic review. BMJ 2001; 322: 406–9.CrossRefGoogle ScholarPubMed
7 Lambert, MJ, Whipple, JL, Hawkins, EJ, Vermeersch, DA, Nielsen, SL, Smart, DW. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Pr 2003; 10: 288301.Google Scholar
8 Gilbody, SM, House, AO, Sheldon, TA. Outcome measures and needs assessment tools for schizophrenia and related disorders. Cochrane Database Syst Rev 2003; issue 1: CD003081.Google Scholar
9 Marshall, S, Haywood, K, Fitzpatrick, R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract 2006; 12: 559–69.Google Scholar
10 Sapyta, J, Riemer, M, Bickman, L. Feedback to clinicians: theory, research, and practice. J Clin Psychol 2005; 61: 145–53.Google Scholar
11 Hedges, LV. Distribution theory for Glass's estimator of effect size and related estimators. J Educ Stat 1981; 6: 107–28.Google Scholar
12 Hedges, LV, Oklin, I. Statistical Methods for Meta-analysis. Academic Press, 1985.Google Scholar
13 Lambert, MJ, Whipple, JL, Vermeersch, DA, Smart, DW, Hawkins, EJ, Nielsen, SL, et al. Enhancing psychotherapy outcomes via providing feedback on client progress: a replication. Clin Psychol Psychoth 2002; 9: 91103.CrossRefGoogle Scholar
14 Hawkins, EJ, Lambert, MJ, Vermeersch, DA, Slade, KL, Tuttle, KC. The therapeutic effects of providing patient progress information to therapists and patients. Psychother Res 2004; 14: 308–27.Google Scholar
15 Lambert, MJ, Whipple, JL, Smart, DW, Vermeersch, DA, Nielsen, SL, Hawkins, EJ. The effects of providing therapists with feedback on patient progress during psychotherapy: are outcomes enhanced? Psychother Res 2001; 11: 4968.CrossRefGoogle ScholarPubMed
16 Bauer, S. Outcome Monitoring and Feedback: Strategies for Optimisation of Inpatient Psychotherapy? (in German). Doctoral Thesis (PhD), Faculty of Social and Behavioural Sciences, University of Tübingen, Germany, 2004.Google Scholar
17 DerSimonian, R, Laird, N. Meta-analysis in clinical trials. Control Clin Trials 1986; 7: 177–88.Google Scholar
18 Higgins, JPT, Thompson, SG, Deeks, JJ, Altmann, GG. Measuring inconsistency in meta-analyses. BMJ 2003; 327: 557–60.CrossRefGoogle ScholarPubMed
19 Egger, M, Davey Smith, G, Schneider, M, Minder, CE. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–34.Google Scholar
20 Trudeau, LS. Effects of a Clinical Feedback System on Client and Therapist Outcomes in a Rural Community Mental Health Center. Doctoral Thesis (PhD), Institute for Social and Behavioral Research, Iowa State University, USA, 2001.Google Scholar
21 Berking, M, Orth, U, Lutz, W. Effects of systematic feedback to the therapist on patient progress. An empirical study in a cognitive-behavioral inpatient setting (in German). Z Kl Psych Psychot 2006; 35: 21–9.Google Scholar
22 Schmidt, U, Landau, S, Pombo-Carril, MG, Bara-Carril, N, Reid, Y, Murray, K, et al. Does personalized feedback improve the outcome of cognitive-behavioural guided self-care in bulimia nervosa? A preliminary randomized controlled trial. Brit J Clin Psychol 2006; 45: 111–21.Google Scholar
23 Brodey, BB, Cuffel, B, McCulloch, J, Tani, S, Maruish, M, Brodey, I, et al. The acceptability and effectiveness of patient-reported assessments and feedback in a managed behavioral healthcare setting. Am J Manag Care 2005; 11: 774–80.Google Scholar
24 Slade, M, McCrone, P, Kuipers, E, Leese, M, Cahill, S, Parabiaghi, A, et al. Use of standardised outcome measures in adult mental health services. Randomised controlled trial. Br J Psychiatry 2006; 189: 330–6.CrossRefGoogle ScholarPubMed
25 Whipple, JL, Lambert, MJ, Vermeersch, DA, Smart, DW, Nielsen, SL, Hawkins, EJ. Improving the effects of psychotherapy: the use of early identification of treatment and problem-solving strategies in routine practice. J Couns Psychol 2003; 50: 5968.Google Scholar
26 Ashaye, OA, Livingston, G, Orrell, MW. Does standardized needs assessment improve the outcome of psychiatric day hospital care for older people? A randomized controlled trial. Aging Ment Health 2003; 7: 195–9.Google Scholar
27 Marshall, M, Lockwood, A, Green, G, Zajac-Roles, G, Roberts, C, Harrison, G. Systematic assessments of need and care planning in severe mental illness: cluster randomised controlled trial. Br J Psychiatry 2004; 185: 163–8.CrossRefGoogle ScholarPubMed
28 Lambert, MJ, Burlingame, GM, Umphress, V J, Hansen, N B, Vermeersch, D A, Clouse, G, et al. The reliability and validity of the Outcome Questionnaire. Clin Psychol Psychoth 1996; 3: 106–16.Google Scholar
29 Becker, T, Vazquez-Barquero, JL. The European perspective of psychiatric reform. Acta Psych Scand 2001; 104: 814.CrossRefGoogle Scholar
30 Lambert, MJ, Hansen, NB, Finch, AE. Patient-focused research: using patient outcome data to enhance treatment effects. J Consult Clin Psych 2001; 69: 159–72.Google Scholar
31 Lueger, RJ, Howard, KI, Martinovich, Z, Lutz, W, Anderson, EE, Grissom, G. Assesssing treatment progress of individual patients using expected treatment response models. J Consult Clin Psych 2001; 69: 150–8.CrossRefGoogle Scholar
32 Kordy, H, Hannöver, W, Richard, M. Computer-assisted feedback-driven quality management for psychotherapy: the Stuttgart-Heidelberg model. J Consult Clin Psych 2001; 69: 173–83.Google Scholar
33 Hatfield, DR, Ogles, BM. The use of outcome measures by psychologists in clinical practice. Prof Psychol Res Pr 2004; 35: 485–91.Google Scholar
34 Gilbody, SM, House, AO, Sheldon, TA. Psychiatrists in the UK do not use outcome measures. National survey. Br J Psychiatry 2002; 180: 101–3.Google Scholar
35 Jacobson, NS, Truax, P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psych 1991; 59: 12–9.CrossRefGoogle ScholarPubMed
Figure 0

Table 1 Study properties

Figure 1

Table 2 Characteristics of the included studies and their participants

Figure 2

Table 3 Feedback systems useda

Figure 3

Fig. 1 Forest plot for short-term mental health outcome (random-effects model).a a. Treatment outcomes were coded as short-term if they were measured within 9 weeks after initial assessment.

Figure 4

Fig. 2 Forest plot for long-term mental health outcome (random-effects model).a a. Treatment outcomes were coded as long-term if they were measured between 3 and 12 months after initial assessment.

Figure 5

Fig. 3 Forest plot for treatment duration (random-effects model).

Figure 6

Table 4 Results of subgroup analyses (random-effects model)

Figure 7

Fig. 4 Funnel plot of estimated effect sizes (Hedges' g) against standard error of the trials for short-term mental health outcome.

Submit a response

eLetters

No eLetters have been published for this article.