Hostname: page-component-7c8c6479df-fqc5m Total loading time: 0 Render date: 2024-03-28T09:11:39.681Z Has data issue: false hasContentIssue false

The evidence base and readability of Freeman et al on virtual reality for treating delusions

Published online by Cambridge University Press:  02 January 2018

John L. Fresen*
Affiliation:
IT Services, University of Oxford, Oxford, UK. Email: john.fresen@it.ox.ac.uk
Rights & Permissions [Opens in a new window]

Abstract

Type
Columns
Copyright
Copyright © Royal College of Psychiatrists, 2017 

I commend Freeman et al Reference Freeman, Bradley, Antley, Bourke, De Weever and Evans1 for their innovative research using virtual reality in the treatment of persecutory delusions. The rather ‘soft’ finding – probably valid, but hardly surprising – is: ‘Cognitive therapy using virtual reality could prove highly effective in treating delusions’.

I have concerns about the evidence base of the study, that is the design, the data, the sample and the statistical methodology. Each of these concerns interrogates the validity and reproducibility of the study. Reference Gardenier and Resnik2,Reference McNutt3

First, the sample size is extremely small – 30 participants. The consequences of this include overestimates of effect size and low reproducibility of results.

Second, 27 of the 30 participants were unemployed. There is little point to any research if one cannot extrapolate from one's sample to some broader reference population. For this, the sample should mimic the population in important ways. However, there is no discussion about the sampling, or the reference population to which extrapolation might be extended.

Third, neither the patients nor the researchers were masked to the randomisation allocations. This, surely, is a fundamental flaw of the experiment. A double-blind experiment should be used to ensure impartiality, and avoid bias, such as, for example, the Hawthorne effect. Reference McCarney, Warner, Iliffe, van Haselen, Griffin and Fisher4

Fourth, the main outcomes – comparing the delusional conviction of the two groups at the beginning and end of testing, as well as their distress – were tested using ANCOVA. But in the results section the authors report: ‘For ratings of conviction in paranoia, a gradual reduction across the scenarios for the threat belief testing group can be seen, whereas the conviction scores remain stable in the exposure group’ (p. ). This suggests that the two groups diverge over time, having different slopes, rather than the assumed homogeneous slopes in the ANCOVA model.

Fifth, the term ‘repeated measures mixed model’ covers a wide range of possible models, and leads one to expect a single model incorporating the repeated measures and random effects, not ten models as are presented in the online supplement. Further, none of the models is clearly articulated in mathematical form.

Sixth, there are no graphs to display the data or statistical results. Tay et al Reference Tay, Parrigon, Huang and LeBreton5 propose the use of graphical descriptives to enhance research rigour, especially in psychology.

It appears that the article is written on two levels. The introduction and method sections, describing participants, design and virtual reality, are clear and lucid. By contrast, the evidence base of the article, discussing the data, models, analysis and results, is almost unintelligible. Further, the small sample size, sampling bias, lack of randomisation masking, lack of model specification and lack of statistical graphics, seriously undermine the study.

The phrase ‘evidence-based research’ has become popular in psychology. Thus, it is incumbent on readers, authors and journal editors to ‘raise the bar’ and demand higher standards of the evidence base of research studies.

References

1 Freeman, D, Bradley, J, Antley, A, Bourke, E, De Weever, N, Evans, N, et al. Virtual reality in the treatment of persecutory delusions: randomised controlled experimental study testing how to reduce delusional conviction. Br J Psychiatry 2016; 209: 62–7.CrossRefGoogle ScholarPubMed
2 Gardenier, JS, Resnik, DB. The misuse of statistics: concepts, tools, and a research agenda. Account Res 2002; 9: 6574.CrossRefGoogle ScholarPubMed
3 McNutt, M. Reproducibility. Science, 2014; 343: 229.CrossRefGoogle ScholarPubMed
4 McCarney, R, Warner, J, Iliffe, S, van Haselen, R, Griffin, M, Fisher, P. The Hawthorne Effect: a randomised, controlled trial. BMC Med Res Method 2007; 7: 30.CrossRefGoogle ScholarPubMed
5 Tay, L, Parrigon, S, Huang, Q, LeBreton, JM. Graphical descriptives: a way to improve data transparency and methodological rigor in psychology. Perspect Psychol Sci 2016; 11: 692701.CrossRefGoogle ScholarPubMed
Submit a response

eLetters

No eLetters have been published for this article.