Hostname: page-component-7c8c6479df-xxrs7 Total loading time: 0 Render date: 2024-03-27T10:35:54.179Z Has data issue: false hasContentIssue false

Psychiatrists in the UK do not use outcomes measures

National survey

Published online by Cambridge University Press:  02 January 2018

Simon M. Gilbody
Affiliation:
NHS Centre for Reviews and Dissemination, University of York, York
Allan O. House
Affiliation:
Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, Leeds
Trevor A. Sheldon
Affiliation:
Department of Health Studies, University of York, York
Rights & Permissions [Opens in a new window]

Abstract

Type
Editorials
Copyright
Copyright © Royal College of Psychiatrists, 2002 

Governmental policy statements on mental health practice over the past decade have emphasised the importance of routinely measuring individual patient outcomes (Department of Health, 1991, 1998; Secretary of State for Health, 1999). Despite the availability of various standardised tools with which to measure the symptom severity of common psychiatric disorders, patient need and wider quality of life and health status, little is known about the actual use of standardised outcomes measures by clinicians (Reference Slade, Thornicroft and GloverSlade et al, 1999). One previous survey of 73 consultant psychiatrists from 1989 established which of a pre-specified range of symptom-based clinical measures were in use at that time (Reference Rice and DonnellyRice & Donnelly, 1992). In view of the central role given to outcomes measurement in the recent National Service Framework for Mental Health (Secretary of State for Health, 1999) we decided to undertake a survey of the current use of outcomes measures in psychiatric practice in the UK.

NATIONAL SURVEY

Questionnaire design

We constructed and piloted a self-completion/self-report questionnaire that sought to identify the routine standardised outcomes measures used by adult psychiatrists for the purpose of:

  1. (a) identifying and assessing the severity of clinical disorders;

  2. (b) identifying patients' needs and deficits in social functioning, and quality of life;

  3. (c) monitoring patient progress;

  4. (d) clinical audit.

Respondents were asked about the use of outcomes measures for the following problems: depression, anxiety and related disorders; schizophrenia and other psychoses; cognitive impairment; drugs and alcohol. Respondents also were asked what outcomes measures their trusts routinely collected or asked them to collect.

Survey method

We conducted a postal questionnaire survey between July and September 2000 of 500 consultant psychiatrists practising in the UK National Health Service (NHS) chosen randomly from UK NHS clinicians responsible for the psychiatric care of working-age adults listed in the Medical Directory.

Questionnaire responses

In total, 369 (74%) of the 500 questionaires were returned, of which 29 were incomplete or ineligible (raw eligible response rate 340/500, 68%; modified response rate 340/471, 72%). Most were general psychiatrists (82%) and worked in non-teaching hospital settings (66%). Details of their responses are given in Table 1.

Table 1 The reported use of standardised measures by clinicians (the percentage values are given with 95% confidence intervals)

Use by clinicians of standardised measures, by problem category Case identification and severity measurement Assessing social functioning, quality of life and needs Measuring clinical change over time Clinical audit
Depression/anxiety
Never 55.3% (49.8-60.1) 80.6% (75.9-84.7) 58.2% (52.8-63.5) 76.5% (71.6-80.9)
Occasionally 34.1% (29.0-39.4) 12.9% (9.6-17.0) 30.5% (25.7-35.8) 15.3% (11.6-19.6)
Routinely 10.5% (7.5-14.4) 6.5% (4.1-9.6) 11.2% (8.0-15.0) 4.1% (2.2-6.8)
Measures used
BDI, 61/340 HoNOS, 20/340 BDI, 49/340 BDI, 18/340
HAD, 53/340 SAS, 9/340 HAD, 41/340 HoNOS, 18/340
HRSD, 46/340 SFQ, 5/340 HRSD, 23/340 HRSD, 13/340
HoNOS, 11/340 GAF, 4/340 HoNOS, 18/340 HAD, 12/340
MADRS, 10/340 CAN, 3/340 MADRS, 10/340
GAF/CGI, 9/340
Schizophrenia/other psychoses
Never 72.9% (67.9-77.6) 75.6% (70.4-79.8) 73.5% (68.5-78.1) 78.8% (74.1-83.0)
Occasionally 20.6% (16.4-25.3) 13.5% (10.1-17.6) 20.0% (15.9-24.7) 11.8% (8.5-15.7)
Routinely 6.5% (4.1-9.6) 11.2% (8.0-15.0) 6.5% (4.1-9.6) 9.4% (6.5-1.3)
Measures used
PANSS, 25/340 PANSS, 20/340 HoNOS, 33/340 HoNOS, 24/340
HoNOS, 20/340 BPRS, 13/340 BPRS, 13/340 PANSS, 6/340
BPRS, 17/340 HoNOS, 16/340 PANSS, 12/340 BPRS, 8/340
Manchester Scale, 9/340 GAF/CGI, 9/340
PSE/SCAN, 6/340
GAF, 5/340
Cognitive impairment
Never 44.7% (39.3-50.2) 83.5% (79.2-87.3) 66.5% (61.2-71.5) 86.5% (82.4-89.9)
Occasionally 40.6% (35.3-46.0) 10.6% (7.5-14.4) 24.7% (20.2-29.6) 10.6% (7.5-14.4)
Routinely 14.7% (11.1-18.9) 5.9% (3.6-8.9) 8.8% (6.0-12.4) 2.9% (1.4-5.3)
Measures used
MMSE, 134/340 HoNOS, 13/340 MMSE, 60/340 MMSE, 13/340
WAIS, 9/340 QL Checklist, 3/340 HoNOS, 13/340 HoNOS, 9/340
WAIS, 6/340
Drugs/alcohol
Never 83.3% (79.1-87.3) 88.8% (84.9-91.9) 91.2% (87.6-94.0) 91.2% (87.6-94.0)
Occasionally 10.6% (7.5-14.3) 5.9% (3.6-8.9) 4.1% (2.3-6.8) 3.5% (1.8-6.1)
Routinely 5.9% (3.6-8.9) 5.3% (3.2-8.2) 4.7% (2.7-7.5) 5.3% (3.2-8.3)
Measures used
CAGE, 10/340 HoNOS, 12/340 HoNOS, 10/340 HoNOS, 8/340
SADQ, 3/340

Case identification and severity measurement

Depression/anxiety and cognitive impairment were the disorders where outcomes measures were used most commonly for identifying and assessing the severity of psychiatric disorders, with around half of clinicians using these measures either routinely or occasionally. The most commonly used measures were the Beck Depression Inventory (Reference Beck and WardBeck & Ward, 1961) (61/340), the Hospital Anxiety and Depression Scale (Reference Zigmond and SnaithZigmond & Snaith, 1983) (53/340) and the Hamilton Rating Scale for Depression (Reference HamiltonHamilton, 1967) (46/340).

The most commonly used measure in detecting cognitive impairment was the Mini-Mental State Examination (Reference Folstein, Folstein and McHughFolsteinet al, 1975). For disorders such as schizophrenia, the majority of consultants (72.9%; 95% CI 67.9-77.6) never used standardised measures. For drug and alcohol problems, the most commonly reported measure was the CAGE questionnaire (Reference Mayfield, Millard and HallMayfieldet al, 1974) (10/340).

Assessing social functioning, quality of life and needs

Only a small minority used a standardised questionnaire for detecting patients' needs and deficits in social functioning and quality of life. The most commonly used was the Health of the Nation Outcome Scale (HoNOS; Reference Curtis, Beevor and WingCurtis & Beevor, 1995). Specific ‘needs assessment’ tools, such as the Medical Research Council (MRC) Needs for Care and the Camberwell Assessment of Needs, rarely were reported as being used in routine practice.

Measuring clinical change over time

Standardised measures most commonly were used in order to measure change over time for depression and anxiety problems, with 19.4% (95% CI 15.3-24.0) of consultants using them either routinely or occasionally. The most commonly reported measures were the Beck Depression Inventory (49/340); the Hospital Anxiety and Depression Scale (41/340) and the Hamilton Rating Scale for Depression (23/340). For those with dementia, the most commonly used questionnaire was the Mini-Mental State Examination (60/340). For other conditions, clinicians rarely used standardised measures.

Clinical audit

Overall, standardised questionnaires were used even less for clinical audit than for the other purposes outlined above. The most commonly reported condition for which they were used was depression/anxiety, where the Beck Depression Inventory was the most common instrument; for other conditions (schizophrenia; drugs and alcohol; and dementia) the HoNOS was used.

Outcomes measures routinely collected by hospitals/trusts

Very few clinicians (46/340, 13.5%; 95% CI 10.0-17.6) reported being required routinely to collect standardised outcomes measures for all patients by their trust. When asked specifically about the HoNOS, 88/340 (26%; 95% CI 21.3-30.1) reported being asked to collect these data for certain patients, and only 28/340 (8.2%; 95% CI 5.5-11.7) reported using specific needs assessment tools (such as the Camberwell Assessment of Need and the MRC Needs for Care).

In contrast to standardised questionnaires, such as the HoNOS, trusts commonly collected routine administrative data, such as length of stay (86.2%; 95% CI 82.7-89.9) and readmission (70.6%; 95% CI 65.4-75.4).

General comments relating to the use and experience of outcomes measurement

Approximately one-third (120/340) used the questionnaire to give comments. Forty respondents expressed a negative view of standardised outcomes measures, questioning their ability to capture the subtlety of multi-faceted outcome and to describe the individual patient. Terms such as “simplistic” and “ pseudo-scientific gloss” were used. One respondent expressed reservation about any scale that “divides continuous fluctuating process into arbitrary categories”. Twenty-eight respondents explicitly questioned the basic psychometric properties of validity, reliability and sensitivity to change for available measures.

Respondents stated that the valid and reproducible use of outcome measures requires a robust infrastructure, particularly in terms of administration and information technology resources, to support the process (n=20). Respondents generally felt that these features were lacking and that this represented a barrier to their use. Additionally, 29 respondents felt that more time and resources would be needed if outcome measurement were to be carried out and used routinely. Twenty-two respondents stated that they did not find the results of standardised outcomes measures particularly useful in clinical practice. One respondent stated that they were more “research tools” rather than instruments that are useful in clinical practice. Another stated that the “use of scales detracts from the therapeutic relationship”.

Comments were largely critical of the HoNOS (n=21) and related to: time to complete (n=16); inadequate psychometric properties (n=8); the lack of value added to routine clinical assessment (n=5); and the lack of enthusiasm among staff (n=7). Positive comments (n=7) included the fact that it could be completed by non-clinicians (n=4) and that it acted as a useful aide mémoire in clinical decision-making (n=3). One person stated that “the HoNOS, although scientifically flawed, is useful for bringing together all members of the multi-disciplinary team”.

IMPLICATIONS FOR MENTAL HEALTH PRACTICE, RESEARCH AND POLICY

Outcomes measurement forms a central component of the National Service Framework, with, for example, the stipulation of a minimum data set (Reference Glover, Knight and MelzerGlover et al, 1997), which includes the HoNOS, to be collected alongside care planning for all of those with severe mental illness and a number of outcomes indicators to be implemented on a routine basis (Secretary of State for Health, 1999). The results of this survey, in particular the barriers identified to outcomes measurement, will be of particular interest to those charged with implementing the National Service Framework for Mental Health. Several areas are worthy of further discussion.

Our main finding is that the majority of psychiatrists do not use outcomes measures in their day-to-day practice. Patient needs and psychosocial problems are measured infrequently in any standardised or consistent way, despite explicit Government policy (Reference Glover, Knight and MelzerGlover et al, 1997; Secretary of State for Health, 1999) to adopt measures such as the HoNOS and needs assessment tools. This may reflect a wider indifference towards and failure to address psychosocial outcomes and needs, or simply a failure to use standardised measures to assess them.

The HoNOS does seem to have found a place in measuring outcome in UK mental health services, albeit a small one. Initial hopes that it would be used on a servicewide basis in order to measure outcome, define population needs and to judge the performance of individual trusts and teams (Reference Wing and DelamontheWing, 1994) have, however, clearly not been realised. Specific concerns expressed by clinicians relating to the time taken to complete the instrument and the poor psychometric properties will need to be addressed if its use is going to be increased. On the other hand, positive aspects of outcomes measures were identified that may be useful in deciding their place in mental health services. In particular, the comments that non-medical personnel can complete instruments such as the HoNOS and that it encouraged discussion within the wider clinical team are worthy of note.

Administrative measures collected by trust — such as length of stay and readmission rates — although easier to collect, may bear little relation to the psychosocial outcome of the individual patient or clinical population. It is administrative outcomes that will form the basis of the performance elements of the NHS (Secretary of State for Health, 1999). Of particular concern is that these figures are the easiest to manipulate or ‘improve’, without conferring any overall health gain on the population or service under consideration (Reference Nutley and SmithNutley & Smith, 1998).

Teams need to be resourced adequately to collect outcomes data, and the instruments that are used must be reasonably concise and easily administered (Reference Slade, Thornicroft and GloverSlade et al, 1999). Additionally, the reluctance among clinicians to collect data on a routine basis may reflect the fact that they see little benefit to themselves or their patients' care. Outcomes measurement is a ‘technology’ that has opportunity costs and therefore must be shown to be beneficial in improving the quality of care as measured by actual outcomes of patients or communities. No direct evidence exists that there is a benefit in this respect for those working and being cared for in either primary care or specialist psychiatric services (Gilbody et al, Reference Gilbody, House and Sheldon2001a ,Reference Gilbody, House and Sheldon b ). Successful implementation of outcomes management will need to overcome the barriers that we identified in this survey of current UK psychiatric practice.

Acknowledgements

S. M. G. was supported by the UK Medical Research Council Health Services Research Training Fellowship Programme. The NHS Centre for Reviews and Dissemination and the Department of Health Studies are part of the Medical Research Council Health Services Research Collaboration (MRC HSRC).

Footnotes

DECLARATION OF INTEREST None.

References

Beck, A. T. & Ward, C. H. (1961) An inventory for measuring depression. Archives of General Psychiatry, 4, 561571.CrossRefGoogle ScholarPubMed
Curtis, R. & Beevor, A. (1995) Health of the Nation Outcome Scales (HoNOS). In Measurement for Mental Health: Contributions from the College Research Unit (ed.Wing, J.), pp. 3346. London: Royal College of Psychiatrists' Research Unit.Google Scholar
Department of Health (1991) The Health of the Nation: A Strategy for England. London: HMSO.Google Scholar
Department of Health (1998) Modernising Mental Health Services: Safe, Sound and Supportive. London: HMSO.Google Scholar
Folstein, M. F., Folstein, S. E. & McHugh, P. R. (1975) ‘Mini-Mental State’: a practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12, 189198.Google Scholar
Gilbody, S. M., House, A. O. & Sheldon, T. A. (2001a) Outcome and needs assessment for schizophrenia and related disorders (Cochrane Review). In The Cochrane Library – Issue I. Oxford: Update Software.Google Scholar
Gilbody, S. M., House, A. O. & Sheldon, T. A. (2001b) Routinely administered questionnaires for depression and anxiety: a systematic review. BMJ, 322, 406409.Google Scholar
Glover, G., Knight, S., Melzer, D., et al (1997) The development of a new minimum data set for specialist mental health care. Health Trends, 29, 4851.Google Scholar
Hamilton, M. (1967) Development of a rating scale for primary depressive illness. British Journal of Social and Clinical Psychology, 6, 278296.Google Scholar
Mayfield, D., Millard, G. & Hall, P. (1974) The CAGE questionnaire. American Journal of Psychiatry, 131, 11211123.CrossRefGoogle ScholarPubMed
Nutley, S. & Smith, P. (1998) League tables for performance improvement in health care. Journal of Health Services Research and Policy, 3, 5057.Google Scholar
Rice, K. & Donnelly, P. (1992) Use of rating scales by consultant psychiatrists. Psychiatric Bulletin, 16, 329330.CrossRefGoogle Scholar
Secretary of State for Health (1999) National Service Framework – Mental Health. London: HMSO.Google Scholar
Slade, M., Thornicroft, G. & Glover, G. S. O. (1999) The feasibility of routine outcome measures in mental health. Social Psychiatry and Psychiatric Epidemiology 34, 243249.Google Scholar
Wing, J. (1994) Measuring mental health outcomes: a perspective from the Royal College of Psychiatrists. In Outcomes into Clinical Practice (ed. Delamonthe, T.), pp. 147152. London: BMJ Publishing.Google Scholar
Zigmond, A. S. & Snaith, R. P. (1983) The Hospital Anxiety and Depression scale. Acta Psychiatrica Scandinavica, 67, 361370.Google Scholar
Figure 0

Table 1 The reported use of standardised measures by clinicians (the percentage values are given with 95% confidence intervals)

Submit a response

eLetters

No eLetters have been published for this article.