Background Little research on the value of Health of the Nation Outcome Scales (HoNOS) has occurred in out-patient settings, particularly psychotherapy services.
Aims To determine whether HoNOS provides an adequate assessment for psychotherapy services which is sensitive to change.
Methods HoNOS ratings from 1688 patients from eight out-patient psychotherapy services were collected. Of these, 362 also had ratings post-treatment. Mean scores, pre-to post-treatment differences, and reliable and clinically significant change criteria were calculated for HoNOS items and for total scores.
Results The mean total HoNOS rating was 8.93, which is comparable to psychiatric out-patients. Only three items showed sufficient variability to use in assessing pre- to post-treatment change.
Conclusions Significant limitations were found in rating items that commonly present to psychotherapists. The lack of variability in most items limits HoNOS's usefulness in this population.
The Health of the Nation Outcome Scales (HoNOS; Wing et al, 1998) were developed to measure health outcomes in response to the then UK government's target to “improve significantly the health and social functioning of mentally ill people” (Department of Health, 1992). HoNOS was developed to become a standardised assessment tool to be used routinely by all mental health practitioners (Wing et al, 1996). Indeed, HoNOS has recently been formally adopted for use with major Care Programme Approach reviews, and one-third of English trusts surveyed between October 1997 and May 1998 were using HoNOS routinely in one or more service settings (Wing et al, 2000).
HoNOS and psychological therapy services
Field trials in the development of HoNOS focused on adults with severe mental illness attending in-patient and community psychiatry services. The developers of HoNOS state that it is acceptable, clinically useful, reliable, sensitive to change and useful for administration and planning within such settings (Wing et al, 1998). Recent independent studies, however, have questioned its reliability (Orrell et al, 1999), sub-scale structure (Trauer, 1999), sensitivity to change (Trauer et al, 1999) and appropriateness for routine clinical use in busy psychiatry services (Bebbington et al, 1999; Sharma et al, 1999). Versions of HoNOS have been developed for the specialities of child psychiatry (HoNOS for Children and Adolescents: Gowers et al, 1999) and psychiatry services for the elderly (HoNOS 65+: Burns et al, 1999), but there is currently no specific version of HoNOS for use in psychotherapy and psychological treatment services.
The above would suggest the need for more evidence on the performance of HoNOS within psychotherapy and psychological treatment settings before it is espoused as the standard outcome measure for routine use by all mental health practitioners. Three specific questions arise. First, does HoNOS provide an adequate baseline assessment of psychotherapy patients' problems? Second, is HoNOS sensitive to changes following treatment? And third, is HoNOS appropriate for informing and guiding service development? These questions were addressed via practitioners belonging to a practice research network (PRN) under the auspices of the Society for Psychotherapy Research (SPR).
The SPR PRN is a multi-site initiative involving a group of psychotherapy practitioners and researchers in the north of England. Network practitioners collaborate to collect, share and utilise standardised clinical and service effectiveness information as part of routine practice (Audin et al, 2001). SPR PRN practitioners incorporated HoNOS into their evaluation instrumentation to test its feasibility for routine use in out-patient and community psychotherapy and psychological treatment services.
The SPR PRN consists of eight National Health Service (NHS) secondary and tertiary out-patient and community psychotherapy services based in Manchester, Bradford, Leeds, Wakefield, York, Liverpool, Preston and Sheffield. The SPR PRN is coordinated by the Psychological Therapies Research Centre at the University of Leeds, which is responsible for analysis and reporting of network data.
The SPR PRN uses a broad definition of psychotherapy as recommended by the NHS Executive's review of NHS psychotherapy services in England (Parry & Richardson, 1996). Psychotherapy covers tertiary specialist services, those linked to community mental health teams and those linked directly to primary care. The term also covers a wide range of modalities used by constituent practitioners, including psychodynamic—`interpersonal, cognitive—behavioural, systemic and integrative.
The mean age of the patient sample (n=1688) was 35.7 years (s.d.=0.6 years) and 60% were female. Almost a quarter (24%) of patients were unemployed and 28% were living alone at the time of the assessment. Almost one-third (31%) of patients assessed had previously received psychological treatment within a secondary-level community service. Over half the patient sample (53%) were referred by a general practitioner and over a quarter (26%) by a psychiatrist. The average waiting time from referral to assessment was 12 weeks.
Health of the Nation Outcome Scales
The 12 HoNOS items are each scored 0-4, yielding a total score in the range 0-48 (Wing et al, 1996). Item 8 (see Appendix) of HoNOS requires that practitioners identify one other mental and behavioural problem from a list of 10: phobic, anxiety, obsessive—compulsive, stress, dissociative, somatoform, eating, sleep, sexual, and other. This list appeared particularly relevant to the psychotherapy patient population but SPR PRN practitioners felt limited by the necessity to identify only one problem. We therefore amended HoNOS to allow practitioners to rate the main other mental and behavioural problem but, in addition, also to identify up to four supplementary problems from the list.
Representatives from each service attended a HoNOS training session provided by staff from the Research Unit of the Royal College of Psychiatrists. The representatives then shared their training information with their colleagues in each service. All participating therapists were also provided with a HoNOS glossary containing rating instructions. Practitioners completed HoNOS at two time points: at first assessment and at discharge. HoNOS was completed at first assessment to assess its suitability as a profiling tool, and at discharge to test its ability to provide outcome information.
HoNOS data sets
Within the eight participating NHS out-patient and community psychotherapy services, 65 psychotherapists provided HoNOS assessment data and 55 therapists provided discharge data. The same therapist rated HoNOS at assessment and discharge for 239 (66%) cases. Assessment data were recorded for 1688 patients, with HoNOS being fully complete for 76% of these cases. A total of 1166 patients were accepted for therapy, and discharge forms were completed for 362 (31%) of these patients. Fully complete pre- and post-HoNOS data were recorded for 208 patients (18%). A further 85 patients with discharge data only were not included in analysis.
Additional clinical and service information
HoNOS was incorporated into an existing evaluation protocol, designed by the network practitioners to gather clinical and service data. At assessment, additional information regarding waiting times from referral to assessment, patient demographics, previous and current therapy, reason for referral (qualitative data) and assessment outcome was also recorded. At discharge, additional data concerning waiting times for therapy, number of sessions attended, therapy type, modality and frequency, therapy ending, benefits of therapy (qualitative data) and follow-up arrangements were collected. These data are not reported in detail in this paper.
Practitioners completed HoNOS after an average of one assessment. Figure 1 shows the distribution of ratings for each HoNOS item at assessment using boxplots (Norusis, 1992). (For reference, HoNOS item descriptors are given in the Appendix.)
The data shown from HoNOS are not ideal for this form of display, as there are no intermediate values between the integers. Ratings for items 3, 4, 5, 6 and 11 are extremely limited, with all ratings other than 0 identified as outliers. Ratings for items 1, 2, 10 and 12 are limited, having a median rating of 0 and a highest rating of 2, excluding outliers, and could be used only for very limited purposes. Items 7, 8 and 9, however, show reasonable distributions, with the middle of 50% of ratings ranging from 1 to 3, and having lowest and highest values of 0 and 4 respectively. Because of missing data, n for each item presented in Figure 1 varies. All analyses were repeated only with patients having a fully complete HoNOS (n=1279), but no substantial differences were found by restricting the sample in this way.
Table 1 shows the mean total ratings for the four HoNOS sub-scales, A, B, C and D, and shows a clear rank order with Symptomatic Problems being the highest (owing to the high ratings of items 7 and 8), followed by Social Problems, then Behavioural Problems and finally Impairment. The mean total HoNOS rating was 8.93 (s.d.=5.21).
Table 2 shows frequencies for the Other Mental and Behavioural Problems identified in item 8. The frequencies illustrate that allowing raters to identify only one main Other Mental and Behavioural Problem restricts choice and leads to misleading profile data. When raters are permitted to identify additional problems to reflect the co-presentation of symptoms, the number of patients identified as displaying those problems rises considerably. For example, the number of patients identified as having sleep problems rises from 2.7% when only a single problem can be identified, to 16.6% when raters are permitted to identify co-presenting problems.
Of the 1166 patients accepted for therapy, discharge forms were completed for 362 (31%). In order to establish whether the discharge subsample was significantly different from the full assessment sample, t-tests between the assessment means of both samples were conducted. The assessment mean total score for the discharge subsample was significantly higher than that for the full assessment sample (t=2.28, d.f.=361, P=0.023). Item 2 (t=-2.33, d.f.=346, P=0.021), item 7 (t=-4.21, d.f.=358, P<0.0005), item 8 (t=-2.86, d.f.=327, P=0.004) and item 12 (t=3.27, d.f.=334, P=0.001) showed differences at assessment between the full assessment and discharge populations. This means that the discharge subsample is not necessarily representative of the whole sample, as it excludes patients still in treatment and those seen for assessment only, as well as therapists who fail to complete forms.
Figure 2 shows mean HoNOS item scores at assessment and at discharge for patients having both assessment and discharge ratings. The n for each item varies owing to missing data. Again, all analyses were repeated with cases having complete HoNOS data (all 12 items), and no substantial differences were found by being selective in this way. A reduction in mean scores can be seen for all items, with the greatest reduction seen in items 7, 8 and 9. To reinforce Figure 2, mean assessment and discharge scores are displayed in Table 3 along with change scores, confidence intervals, outcome effect sizes and t-test results. Items 4, 5 and 11 shows the least change. Outcome (i.e. pre—post) effect sizes are a useful means of interpreting the extent of change between assessment and discharge scores, and are calculated by dividing the change score by the standard deviation of the assessment score. Change on the full HoNOS is statistically significant (t=13.18, d.f.=308, P<0.0005) with a moderate pre—post effect size of 0.69 standard deviation units. Only items 4, 5 and 11 showed change scores that were not significant at the P<0.05 level.
Assessment and discharge total scores for each patient can also be summarised in terms of reliable and clinically significant change (Evans et al, 1998). Reliable change (i.e. change that is not due to chance or measurement error) was calculated as any change greater than 1.96 times the standard error of difference. The threshold for clinical change was calculated as the mean total assessment score plus the mean total discharge score, halved. More sophisticated methods of determining clinical cut-off points which take into account clinical and non-clinical norms are available (Evans et al, 1998), but lack of HoNOS normative data in non-clinical and well populations prevented use of such methods.
Table 4 summarises the reliable and clinically significant change results. For over half the sample (52%), HoNOS ratings did not change to either a statistically reliable or a clinically significant extent. Almost a quarter (24%) of patients' scores reduced to an extent that indicated both reliable and clinically significant improvement, and only 2% showed both reliable and clinically significant deterioration.
Does HoNOS provide an adequate assessment of psychotherapy patients' problems?
In response to this first question, our results suggest that the answer is at best equivocal and at worst negative. Analysis of HoNOS assessment ratings suggests that this instrument does not provide adequate coverage of the range of problems presented by psychotherapy patients. Extremely low assessment ratings for all items except items 7, 8 and 9 indicate that the majority of HoNOS items are irrelevant and inappropriate for an out-patient psychotherapy population. This reflects the overriding importance of HoNOS being useful in the enduring, severe mental illness category. These patients are likely to have complex care plans, reflecting difficulties in social and occupational functioning, although the severity of disorder itself may be comparable to that of out-patients treated in psychotherapy services.
Conversely, important problem areas such as self-esteem and post-traumatic stress disorder, which are highly relevant to psychotherapy patients, are not included. Items 7, 8 and 9 (Problems with Depressed Mood, Other Mental and Behavioural Problems, and Problems with Relationships) do provide a reasonable level of profiling at assessment, with the Other Mental and Behavioural Problems in item 8 being particularly pertinent to psychotherapy patients. However, item 8 results show that there is a clear need for psychotherapists to be able to record co-presenting problems, rather than only the ‘ main’ problem, as in the current HoNOS guidelines. When practitioners are able to identify more than one problem from the list of 10, item 8 becomes much more useful for assessment profiling. Anxiety was the most prevalent problem in item 8, being identified as an issue for almost half the sample (45.8%). It could be argued that anxiety should be a separate item which practitioners could rate more fully on the 0-4 scale. The inadequacy of item 8 to record sufficient detail regarding Other Mental and Behavioural Problems has also been reported in a recent validation study on HoNOS with adult psychiatric patients (McClelland et al, 2000).
The high numbers of missing data further limit the ability of HoNOS to provide assessment profile information. Missing data indicate that the information required for the rating was either not known, not applicable, or that the practitioner was unsure how to rate because of difficulty in understanding the guidelines. The HoNOS guidelines may be a contributing factor to the low ratings, as the wording leans towards definitions drawn from patients who are likely to have complex care plans. The definition of problems relating to occupation and other activities in item 10, for example, is worded in such a way that few patients treated in out-patient settings might be expected to meet the criteria for having even mild problems in this area.
Is HoNOS sensitive to changes following treatment?
In response to the second question, our results suggest that HoNOS is severely limited in its ability to detect and record clinically meaningful change. Its insensitivity to change is evident for the majority of items, with the exception of items 7, 8 and 9. The minimal change for most items is due to low assessment ratings, leaving little scope for a further reduction in rating. Items 7, 8 and 9 do appear to be useful for identifying change, as shown by high effect sizes and t-values in Table 3. The value of the total score for showing change is uncertain. There is sufficient variability for the measure to be used, but cut-off scores between clinical and non-clinical reference populations are not available.
In terms of placing the present sample in comparison with other samples reported in the literature, the mean shown for the present sample can be compared to mean totals from other studies with psychiatric populations. A mean of 9.98 was reported in the original reference sample (Wing et al, 1998). Information is also available by diagnostic group, ranging from 8.44 for alcohol/drug-related disorders to 12.81 for borderline personality disorder in the Victorian field trial (Trauer et al, 1999). In the Camberwell evaluation of severe mental illness the mean total was 12.0 (Slade et al, 1999). However, in out-patient community samples, much lower means of 8.5 (Orrell et al, 1999) and 8.13 (McClelland et al, 2000) have been found. Hence, the mean value found in the psychotherapy and psychological treatment settings sampled in the present study was directly comparable to samples of psychiatric patients treated in the community but lower than the mean for samples with severe mental illness.
Is HoNOS appropriate for informing and guiding service development?
Regarding the third question, our findings suggest that ‘the jury is still out’. The use of HoNOS by the practitioners in the SPR PRN identified several limitations in its suitability for out-patient psychotherapy and psychological treatment services, particularly regarding its profiling ability and sensitivity to change. Consequently, it is limited in its capacity to inform and guide service delivery, for example, in prioritising referrals or aiding discharge. Ultimately, the suitability of HoNOS in its current form for informing psychotherapy service development is questionable.
Limitations of the current study
The current study was limited in that, while all practitioners received basic training in HoNOS completion, the level of interrater reliability was not formally established. The HoNOS field trials used mainly nurses and psychiatrists to test interrater reliability, with clinical psychologists accounting for 1.5% of those practitioners included. Psychotherapists were not included in the trials (Wing et al, 1999). Establishing interrater reliability would have provided useful information both on the reliability of the psychotherapists' ratings for this study, and as a sample of the profession more widely.
However, obtaining interrater reliability data is not particularly compatible with using HoNOS in everyday clinical practice. First, its utility is going to have to be accepted — or not — on the basis of how it is most likely to be used in routine practice. Second, the use of a second standardised measure alongside HoNOS would have provided useful information on the concurrent validity of HoNOS in psychotherapy populations, as well as providing comparison profile and change data. Third, data attrition meant that the discharge data sample was restricted in size to less than a third (31%) of patients accepted for therapy. Reasons for the low return of discharge forms include therapist forgetfulness, practicalities of completing the forms in busy practice settings, patients not attending or dropping out of therapy, long waiting times and long therapy contracts. Finally, time taken to complete HoNOS was not formally recorded in the current study but would have provided additional information on the appropriateness of HoNOS for routine use in psychotherapy settings.
Implications for the development of HoNOS
If HoNOS is to be taken on by psychotherapy services, further tests of its suitability need to be conducted. The limited range of field trials to date, focusing mainly on patients with severe mental illness, is recognised by the developers (Wing et al, 1998). The SPR PRN would welcome such trials, but would also strongly encourage the development of a specific psychotherapy version of HoNOS, or an alternative measure which taps into a wider range of psychological difficulties to allow accurate profiling and outcomes measurement for psychotherapy patients. As an interim measure we would recommend recording multiple responses under item 8 (Other Mental and Behavioural Problems), as recording only single items markedly reduces the apparent frequency of common presenting problems. We conclude that it is possible to record such supplementary data without altering the main instrument.
Overactive, aggressive, disruptive or agitated behaviour
Problem-drinking or drug-taking
4 Cognitive problems
5 Physical illness or disability problems
6 Problems associated with hallucinations and delusions
7 Problems with depressed mood
8 Other mental and behavioural problems
9 Problems with relationships
10 Problems with activities of daily living
11 Problems with living conditions
12 Problems with occupation and activities
(0, no problem; 1, minor problem requiring no action; 2, mild problem but definitely present; 3, moderately severe problem; 4, severe to very severe problem)
Clinical Implications and Limitations
For psychiatrists practising psychotherapy, Health of the Nation Outcome Scales (HoNOS) have restricted sensitivity as an instrument for sole use in routine practice.
The restrictions imposed by the wording of item 8 of HoNOS can be ameliorated by allowing practitioners to rate the presence of additional symptoms and problems.
Reliable and clinically significant change can be assessed, but only the total scores have enough variability to be used routinely.
No ratings of interrater reliability were made.
High levels of data attrition occurred at follow-up.
The use of integers on a 0-4 scale restricts the analyses possible at the item level.
Thanks are due to practitioners of the Society for Psychotherapy Research (Northern) UK Practice Research Network.
- Received May 22, 2000.
- Revision received November 3, 2000.
- Accepted November 20, 2000.
- © 2001 Royal College of Psychiatrists