The British Journal of Psychiatry
Service quality and clinical outcomes: an example from mental health rehabilitation services in England
Helen Killaspy, Louise Marston, Rumana Z. Omar, Nicholas Green, Isobel Harrison, Melanie Lean, Frank Holloway, Tom Craig, Gerard Leavey, Michael King
  • Declaration of interest

    None.

Abstract

Background

Current health policy assumes better quality services lead to better outcomes.

Aims

To investigate the relationship between quality of mental health rehabilitation services in England, local deprivation, service user characteristics and clinical outcomes.

Method

Standardised tools were used to assess the quality of mental health rehabilitation units and service users’ autonomy, quality of life, experiences of care and ratings of the therapeutic milieu. Multiple level modelling investigated relationships between service quality, service user characteristics and outcomes.

Results

A total of 52/60 (87%) National Health Service trusts participated, comprising 133 units and 739 service users. All aspects of service quality were positively associated with service users’ autonomy, experiences of care and therapeutic milieu, but there was no association with quality of life.

Conclusions

Quality of care is linked to better clinical outcomes in people with complex and longer-term mental health problems. Thus, investing in quality is likely to show real clinical gains.

Recent health policy assumes that better quality services produce better clinical outcomes. Since 2004, this assumption has financially incentivised the delivery of primary care for chronic medical conditions in England through the Quality and Outcomes Framework.1 Although the same approach is now being encouraged for other healthcare systems, including mental health,2 the relationship between service quality and clinical outcome remains largely unexplored. One large study that evaluated the impact of the Quality and Outcomes Framework on diabetes care found no clear association with improved clinical outcomes over the 3 years before and after its introduction.3 However, we cannot conclude from one study that quality and outcome are not related. Service quality is a complex, multidimensional construct that extends beyond the delivery of specific evidence-based treatments and interventions, making it difficult to operationalise and measure. Recent government policy2 has adopted Lord Darzi’s definition of quality as incorporating the effectiveness and safety of treatment and care alongside a positive experience for those who use services.4 A specific, validated quality assessment tool that incorporates these dimensions has been developed for mental health rehabiliation services, the Quality Indicator for Rehabilitative Care (QuIRC).5,6

Mental health rehabilitation services provide specialist, tertiary care to people with complex problems who have not recovered adequately from an acute episode of illness to return home. At any time, about 1% of people with schizophrenia are in receipt of in-patient rehabilitation.7 The cost of services that provide for this group is between 25 and 50% of the total national mental health budget in England.8 In other words, they are a low volume, high need, high cost group with complex problems that complicate their recovery. These include treatment resistance, which occurs in up to 30% of people with schizophrenia,9 cognitive impairment, pervasive negative symptoms,1012 poor social functioning13 and challenging behaviours.7,14 Rehabilitation services in England aim to stabilise service users’ symptoms, maximise social functioning and promote autonomy to facilitate successful community discharge.15 However, there have been no published studies of the outcomes for users of contemporary mental health rehabilitation services. This paper reports on the first phase of a major programme of research into these services (the REAL study: Rehabilitation Effectiveness for Activities for Life) that had four objectives:

  1. to determine the current provision and quality of mental health rehabilitation services in England;

  2. to describe service user characteristics;

  3. to investigate whether the quality of the unit was related to service user characteristics and the psychiatric morbidity of the local area;

  4. to investigate whether service user outcomes were related to the quality of the unit.

Method

The research was approved by the South East Essex Research Ethics Committee (Ref. 09/H1102/45) and began in April 2009. All National Health Service (NHS) mental health trusts in England were contacted to confirm whether they had an in-patient or community mental health rehabilitation unit that accepted patients referred from acute admission wards. Units designated as ‘continuing care’, ‘forensic’ or ‘secure rehabiliation’ were excluded. The research was conducted in keeping with usual research governance guidance and local approvals were gained at each site. Data were collected between July 2009 and March 2011.

The quality of each unit was assessed using the QuIRC,5,6 a web-based toolkit, completed by service managers (available at www.quirc.eu), which reports on seven domains of care in longer-term units for people with complex mental health problems (Living Environment; Therapeutic Environment; Treatments and Interventions; Self-Management and Autonomy; Social Inclusion; Human Rights; Recovery-Based Practice). Its content was derived from a systematic literature review of the effectiveness of components of care for this group,16 a review of relevant international care standards, and Delphi exercises with service users, practitioners, carers and advocates.17 The QuIRC has excellent interrater reliability6 and good internal validity.7 It takes about 45 min to complete and comprises 145 questions about: service provision (e.g. number of beds, average length of stay, built environment, treatments and interventions, staffing, staff turnover, training, supervision and disciplinaries); links with community organisations (e.g. colleges, employment agencies, sport and leisure facilities); the therapeutic milieu and recovery-based practices (e.g. collaborative care planning, service user involvement, promotion of service users’ independent living skills); and the protection of service users’ human rights (e.g. their privacy and dignity, their legal rights and the use of restraint and seclusion). Domain scores are calculated from scores on 86 items, the rest providing descriptive data. More details of its content are given in online Table DS1.

Additional descriptive data were gathered from unit managers on: the unit’s assessment procedure; the number of serious incidents in the preceding 12 months; provision of local community rehabilitation services; provision of local supported accommodation; the number of service users discharged to different types of unsupported/supported accommodation in the past 12 months locally and elsewhere; and involvement of consultant rehabilitation psychiatrists in funding/placement panels for service users requiring supported accommodation and in reviews of service users placed outside the local area. Service user characteristics and medication prescribed were collected from unit managers in an anonymised aggregate form (age, gender, diagnosis).

A rating of the psychiatric morbidity of the area in which the unit is situated was also made using the Mental Illness Needs Index (MINI)18 relevant to the postcode of each unit.

Service users who gave informed consent participated in a research interview which took about 30 min. Sociodemographic details, diagnosis, service contact and risk history were corroborated form the case notes. Autonomy was assessed using the Resident Choice Scale (RCS);19 the service user rates the degree to which they have choice over 22 aspects of daily activities and the running of the unit on a four-point scale (‘I have no choice at all about this’, ‘I have very little choice about this’, ‘I can express a choice about this but I do not have the final say’, ‘I have complete choice about this’). The RCS has a maximum possible score of 88. Quality of life was assessed using the Manchester Short Assessment of Quality of Life (MANSA);20 the service user rates 12 aspects of their life on a scale from 1 (couldn’t be worse) to 7 (couldn’t be better) and a total mean score between 1 and 7 is generated. Their Experiences of Care were assessed using the Your Treatment and Care (YTC)21 questionnaire, which has been used in the UK in service user-led assessments of mental health services. The service user rates 25 items related to their care (e.g. ‘I know who my doctor is’) as ‘yes’, ‘no’ or ‘don’t know’. The total number of ‘yes’ answers is summed giving a maximum possible score of 25. Service users’ views on the unit’s therapeutic milieu were assessed using the General Milieu Index (GMI).22 Service users rate their general satisfaction with the unit, with staff and other residents, and the degree to which they feel the unit facilitates their confidence and abilities on a scale of 1 to 5 (from ‘not at all’ to ‘very much’). An assessment of service user function was also made by the researcher using the Global Assessment of Functioning (GAF)23 in order to take this into account as a potential mediator between service quality and clinical outcomes. All service user interviews were completed within a month of the unit manager interview.

Data analyses

Data were analysed using Stata version 11 for Windows. Descriptive data reporting on our first two objectives are presented as frequencies and percentages or means and standard deviations (s.d.) as appropriate.

With regard to our third objective, multiple linear regression was used to investigate which covariates were associated with unit quality (QuIRC domain scores). Our sample size (133 units) allowed us to estimate up to 8 coefficients in each model, using the rule of 15 observations per coefficient estimated to achieve adequate precision.24 Covariates selected a priori were: location of unit (hospital or community) – units within hospital grounds were recategorised as community as they have been previously found to be more similar in profile to community-based units than hospital wards;6 psychiatric morbidity of the area local to the unit; percentage of male service users; mean age of service users; service users’ mean GAF score; and percentage of service users detained involuntarily.

Our fourth objective was to investigate whether unit quality (QuIRC domain scores) was associated with service user outcomes, namely autonomy (RCS), quality of life (MANSA), experiences of care (YTC) and therapeutic milieu (GMI). Our sample included 739 service users, with 616 service users having complete data on all variables. Using an intracluster correlation of 0.04, an average cluster size of 14 and the rule of 15, this allowed us to estimate up to 27 coeffcients with adequate precision in each model. As YTC data were left skewed, they were categorised into tertiles and analysed using the proportional odds regression with robust standard errors to account for clustering within unit. The assumption of proportional odds required by the model (that the odds ratio comparing one level of a particular covariate with another level was constant across all categories of YTC) was satisfied by the data. Other outcomes were analysed using linear marginal models based on generalised estimating equations.25 All variables (covariates and outcomes) except the MANSA had only a small proportion of missing values. To deal with the missing values in the MANSA data, a binary variable was created by allocating 1 to any service user with a missing MANSA value and 0 otherwise, and a marginal logistic regression model was used to identify the predictors of missingness. These were included in the regression model for MANSA and a complete case analysis was performed controlling for psychiatric morbidity of the area local to the unit (MINI score). Results of these analyses are expressed per 10 percentage point change in QuIRC domain score. As part of the secondary analyses, we also used multiple imputation based on chained equations26 to estimate the missing MANSA values accounting for clustering of service users within units.

Results

Of the 60 NHS mental health trusts in operation in England in 2009, all had at least one in-patient (or community-based equivalent) mental health rehabilitation unit. Three trusts declined to participate in the study, two failed to complete local research governance approval within the Phase 1 study period and therefore could not be included and three trusts closed their rehabiliation units prior to participation. The response rate was therefore 87% (52/60). A total of 133 units were identified, with a median 2 units per trust (interquartile range 1–2). These units had a total of 1809 beds, of which 1647 (91%) were occupied. Forty-three service users (3%) occupied beds that were designated respite or continuing care rather than rehabilitation and were therefore excluded from the study. Of the remaining 1604 service users, 129 (8%) lacked capacity to give informed consent to participate in a research interview, 20 (1%) lacked adequate English to participate, 108 (7%) were unavailable for interview as they were on planned leave elsewhere and 12 (<1%) were unavailable as they were absent from the unit without leave. Of the remaining 1335 service users, 192 (14%) were not interviewed as they were off the unit whenever the researchers visited, 404 (30%) declined to participate and 739 (55%) were interviewed. The response rates of service users across units ranged from 40% to 100% but were not found to have any association with unit quality (QuIRC domain scores).

Unit characteristics

Unit characteristics are shown in Table 1 (for full details see online Table DS2). The majority were based in the community in suburban areas, with a mean of 14 beds and a mean of 16 admissions in the previous year. Most service users were admitted from acute admission wards or directly from the community, with a small number coming from secure settings. Overall, 124 units (93%) provided single bedrooms only and 85% provided separate women-only and mixed-gender communal areas. Overall, 42% of units used standardised measures in their assessment process and 89% used them routinely after admission. All units had input from a psychiatrist and all units were staffed by nurses and support workers, but 21 (17%) had no access to a clinical psychologist and 13 (10%) had no access to an occupational therapist. The total mean staff per unit was 21 (s.d. = 6) and a mean of 20% (s.d. = 15) had left in the previous 2 years. The mean staff to service user ratio was 1.58 (s.d. = 0.47).

View this table:
Table 1

Mental health rehabilitation unit characteristics

On average, 1 (s.d. = 2) member of staff was trained in family psychoeducation and 1 (s.d. = 2) patient was receiving this intervention, 1 (s.d. = 2) member of staff was trained in cognitive–behavioural therapy (CBT) and 2 (s.d. = 3) patients were receiving it. Most unit managers (85%) reported that their service users usually received fewer than ten CBT sessions. All units used individualised care plans and 126 (95%) provided individualised programmes of activities. Most unit managers (95%) reported that their service had links with local community sports facilities. A wide range of other community links were reported; 74% with local churches or other religious organisations, 61% with local entertainment venues such as cinemas, 53% with local cafes, and 20% with other community organisations. In 18% of units, at least one service user was reported to be attending a mainstream employment scheme and on average 1 (s.d. = 1) service user was attending a local college.

On average, 70% (s.d. = 22) of service users in each unit were prescribed atypical antipsychotic medication. Very few were prescribed more than two antipsychotic medications (25 service users across all units) and a mean of 33% (s.d. = 20) were prescribed clozapine. There were few serious incidents (n = 35) or staff disciplinaries (n = 21) in the preceding 12 months across all units. Of those unit managers that answered the question, 30/96 (31%) reported that the consultant psychiatrist was involved in agreeing funding of out-of-area placements and 58/98 (59%) reported that they were involved in reviewing people placed outside the local area. Half the unit managers who answered the question reported the service had a local community rehabilitation team (53/104, 51%).

Service user characteristics

The majority of service users were White men, with a mean age of 40 years, a median 13-year history of contact with mental health services, and four previous admissions. The majority had a diagnosis of schizophrenia or schizoaffective disorder (81%). The median length of the current admission was 18 months and a third of service users were currently detained involuntarily. Almost half had a history of self-neglect or self-harm and over half had a history of assault on others. There were very high levels of satisfaction with care (YTC) and the average GAF score suggested moderate levels of symptoms and impairment of social and occupational functioning (Table 2; see online Table DS3 for full details).

View this table:
Table 2

Service user characteristics

Factors associated with unit quality

Table 3 (full results shown in online Table DS4) shows that the mean age of service users in a unit was associated with scores in five of the seven QuIRC domains, increasing age being associated with decreasing scores. The largest reduction was in the social inclusion domain where it decreased by 0.37 percentage points for each year of mean age (95% CI –0.64 to –0.10). The percentage of service users detained involuntarily was also associated with a decrease in QuIRC domain score in four domains. The largest reduction was in the Self-Management and Autonomy domain which reduced by 0.12 percentage points for each percentage point increase in those detained (95% CI –0.17 to –0.06). The percentage of men per unit was negatively associated with the Social Inclusion and Therapeutic Environment domains, having a greater influence on the former where for each percentage point increase in male service users the Social Inclusion domain score reduced by 0.11 percentage points (95% CI –0.20 to –0.03). For each point increase in MINI, the Living Environment domain score decreased by 6.78 percentage points (95% CI –11.09 to –2.47). Units located in the community had a Therapeutic Environment score 3.58 percentage points higher (95% CI 0.11 to 7.05) than hospital-based units after adjusting for other variables in the model.

View this table:
Table 3

Association between unit quality (Quality Indicator of Rehabilitative Care domain scores), unit location, service user characteristics and psychiatric morbidity of local area

Unit quality and clinical outcomes

Most QuIRC domains were positively associated with experiences of care (YTC) (Table 4; full results shown in online Table DS5). For example, a 10 percentage point increase in the Treatments and Interventions domain score, resulted in an odds ratio of 1.56 (95% CI 1.17–2.08) for scoring in the highest tertile on the YTC compared with the lower two tertiles. All QuIRC domains were positively associated with autonomy (RCS). The largest of these associations was for the Therapeutic Environment domain where a ten percentage point increase was associated with an increase in the RCS of 3.43 (95% CI 2.04–4.81) points. All QuIRC domains were also associated with service users’ ratings of the units’ therapeutic milieu (GMI). Here the QuIRC domain with the strongest influence was Therapeutic Environment, where a ten percentage point increase was associated with an increase in GMI of 1.18 (95% CI 0.61–1.75). The quality of life scores (MANSA) appeared to be associated with Living Environment and Self-Management and Autonomy, where a ten percentage point increase was associated with very small increases in MANSA scores. Repeating this analysis using imputed MANSA data gave similar results.

View this table:
Table 4

Association between unit quality (Quality Indicator of Rehabilitative Care domain scores) and service user outcomes

Discussion

This paper reports on the first in-depth study of NHS mental health rehabilitation units in England. The high participation rate strengthens the generalisability of our findings. However, we acknoweldge that there are many rehabiliation units within the independent sector who could not be included in this study owing to the limitations of our resources, and we did not include units that were designated as ‘forensic’ or ‘secure’ rehabilitation. We found that all NHS trusts had at least one rehabilitation unit catering for a group with particularly complex needs; the relatively high levels of clozapine prescription and extended length of admission corroborate previous descriptions of the treatment resistant nature of illness in this service user group.6 Around half required assistance with some or most activities and one third were difficult to engage in activities.

Although the recommended range of supported accommodation needed to help individuals move on to more independent living27 was usually provided, ‘delayed discharges’ affected about 14% of service users, suggesting inadequate provision of local supported accommodation and ‘silting’ of the care pathway. Service users were discharged only rarely from a rehabiliation unit to an out-of-area placement, but the local psychiatrist with responsibility for the rehabilitation service was commonly not represented on the local placement panel. Current guidance emphasises the importance of this involvement in ensuring the appropriate placement of service users in facilities that are tailored to their needs, that opportunities for local treatment and support have been fully explored prior to a placement being made out of area, and that there is ongoing review of an individual’s suitability for local repatriation at the earliest opportunity.27,28 Our results suggest this guidance is inadequately followed.

Factors associated with quality

Units with a higher proportion of older patients, male patients and patients detained involuntarily were of poorer quality, although the degree of association between these service user characteristics and service quality was small. However, the psychiatric morbidity of the local area had a greater impact on service quality than the nature of the users placed there, although it only influenced one quality domain, the Living Environment. Since areas with greater deprivation tend to have higher levels of psychiatric morbidity,18 there may be a greater pressure on resources in these areas with less available to invest in the built environment. Community-based units appeared to fare better than in-patient units on only one domain of quality (Therapeutic Environment), corroborating previous reports of the less ‘institutional’ culture of non-hospital settings.29 However, our findings suggest that in all other regards, the rehabilitation delivered in hospital-based units was of a similar quality to that in community-based units.

Most unit managers reported high levels of participation in activities on and off the unit, despite around a third of service users being difficult to engage. The high level of individualised activity programmes may have facilitated this. However, very few service users were receiving psychological interventions as recommended by the National Institute for Health and Clinical Exellence.30 This is not surprising since few staff were adequately trained to deliver these. In the current economic context, further investment in clincial psychologists seems unlikely, but a greater focus on training and supervising nurses and other staff to deliver psychological interventions has been shown to be possible in some psychosis services,31,32 although problems with sustainability have also been identified.33,34 This might include so-called ‘low intensity’ psychological interventions that, although lacking a strong evidence base, may be easier for service users with complex mental health problems to engage with and assist with addressing some of the comorbidities and other issues that impede progress towards successful community discharge (e.g. anxiety management, relapse prevention, motivational interviewing).

Relationship between service quality and outcome

There were strong indications that the quality of care provided by units was associated with service users’ autonomy, experiences of care and perception of their therapeutic environment, although it was not associated with quality of life. The cross-sectional nature of our data means we cannot be sure of the direction of these associations but they are encouraging. Although it may be difficult to prevent the destructive impact of chronic psychotic illness on quality of life, it seems rehabilitation services are providing a positive experience of care that facilitates individuals’ autonomy. In addition, the quality of life measure we used considers social and community aspects of life that may simply be outside the current experiences of this service user group, who because of the severity of their symptoms and impairments, have been admitted to an in-patient rehabilitation unit. Furthermore, our cross-sectional data could not assess the longitudinal associations between the quality of rehabilitation services and service users’ quality of life. Later phases of the REAL research programme include a randomised controlled trial to assess the efficacy of a staff training programme to increase service users’ engagement in activities, and a naturalistic cohort study to identify the aspects of care associated with improvements in social functioning and successful, sustained discharge from hospital. Both phases will help to identify the components of in-patient mental health rehabilitation that are associated with better clinical outcomes and their cost-effectiveness and thus help to guide future practice and investment in these services.

National service quality benchmarking

Our findings represent the first comprehensive description of NHS mental health rehabilitation services in England and provide national ‘quality benchmarking’ data for these services. We found a positive association between quality of care and clinical outcomes in these services, suggesting that interventions that improve the quality of care provided are likely to promote service users’ autonomy and experiences of care. Since promotion of autonomy is the main goal of rehabilitation,15,27 ongoing investment in these services is needed to continue to deliver high-quality care that promotes service users’ recovery and abilities in order that they can leave hospital and live as independently as possible in the community. Additional investment in the local supported accommodation pathway is therefore also needed to ensure that service users have an appropriate place to move on to when they are ready to leave the rehabilitation unit.

Finally, our study has also shown that collection of comprehensive service quality assessment data on a national scale is possible when a tailored, reliable and well-validated tool is available. The QuIRC has been incorporated into the Royal College of Psychiatrists’ Accreditation for Inpatient Mental Health Services (AIMS), which ensures that ongoing standardised assessment of the quality of these services can continue.

Acknowledgments

We thank our funders, the fundholders, Camden and Islington NHS Foundation Trust, and the staff, participants and local collaborators at each site for their support.

Footnotes

  • Funding

    This study was funded by the National Institute of Health Research though a Programme Grants for Applied Research (RP-PG-0707-10093). R.Z.O. was partially supported by UCLH/UCL who received a proportion of funding from the Department of Health’s NIHR Biomedical Research Centres funding scheme.

  • Received May 4, 2012.
  • Revision received June 20, 2012.
  • Accepted July 23, 2012.

References

View Abstract