REVIEW ARTICLE |
Department of Child and Adolescent Psychiatry, University Medical Centre Utrecht
Faculty of Social Sciences, Department of Methodology and Statistics, Utrecht University
Department of Child and Adolescent Psychiatry, University Medical Centre, The Netherlands
Correspondence: Ms M. Sprong, Department of Child and Adolescent Psychiatry, University Medical Centre, Heidelberglaan 100, HPA01.468, 3508 GA Utrecht, The Netherlands. Email: m.sprong-2{at}umcutrecht.nl
Funding detailed in Acknowledgements
|
|
|---|
Aims To assess the magnitude of the deficit and analyse associated factors.
Method Twenty-nine studies of mentalising in schizophrenia (combined n=1518), published between January 1993 and May 2006, were included to estimate overall effect size. Study descriptors predicted to influence effect size were analysed using weighted regression-analysis techniques. Separate analyses were performed for symptom subgroups and task types.
Results The estimated overall effect size was large and statistically significant (d=1.255, P<0.0001) and was not significantly affected by sample characteristics. All symptom subgroups showed significant mentalising impairment, but participants with symptoms of disorganisation were significantly more impaired than the other subgroups (P<0.01).
Conclusions This meta-analysis showed significant and stable mentalising impairment in schizophrenia. The finding that patients in remission are also impaired favours the notion that mentalising impairment represents a possible trait marker of schizophrenia.
|
|
|---|
|
|
|---|
Types of mentalising tasks
There is a fair amount of agreement on the definition of theory of mind
among researchers. However, this definition is broad, perhaps reflecting the
fact that it is probably not a unitary function. This has led to a wide
variation in the operationalisation of the concept. One of the most frequently
used types of mentalising tasks is the false belief or deception task (e.g.
Frith & Corcoran, 1996;
Corcoran et al, 1997;
Doody et al, 1998;
Mazza et al, 2001).
In a first-order false belief/deception task, the ability to understand that
someone can hold a belief that is different from the actual state of affairs
is assessed. In a second-order false belief/deception task, participants have
to infer the (false) beliefs of one character about the (false) beliefs of a
second character.
A second type of theory of mind task commonly used in schizophrenia research is an intention-inferencing task, in which the ability to infer a character's intentions from information in a short story is assessed (e.g. Sarfati et al, 1997a,b, 1999, 2000; Sarfati & Hardy-Baylé, 1999). A third type of task measures the ability to understand indirect speech, such as in irony, banter, hints and metaphors (e.g. Corcoran et al, 1995; Langdon et al, 2002; Corcoran, 2003; Corcoran & Frith, 2003; Craig et al, 2004). This is based on the notion that for the understanding of indirect speech an understanding of another person's mental state is required (e.g. Sperber & Wilson, 2002). However, Langdon & Coltheart (2004) showed that comprehension of irony and comprehension of metaphors are unrelated and that having an intact theory of mind is a prerequisite for the interpretation of irony but not for the interpretation of metaphors. Therefore, data on the interpretation of metaphors were excluded from this meta-analysis.
A fourth, less commonly used type of theory of mind task in schizophrenia research is the attribution of mental states to animated geometric shapes which interact in a `socially' complex way (Blakemore et al, 2003; Russell et al, 2006). This type of task may not be fully comparable with the other theory of mind tasks because of the higher level of abstraction involved. Finally, in some studies the `eyes' task is used, in which participants have to infer mental states from looking at pictures of eyes (Kington et al, 2000; Russell et al, 2000; Kelemen et al, 2005). This has been referred to as a theory of mind task, but at face value the construct being measured seems to be different from that assessed by the other paradigms, perhaps assessing emotion recognition abilities or empathy rather than theory of mind.
Since there is a serious lack of research on the psychometric properties (including construct validity and criterion validity) of the many different theory of mind tasks that have been developed (Harrington et al, 2005a), it may not be possible to formulate completely objective inclusion criteria regarding the type of tasks used in the studies. In this meta-analysis this problem is addressed statistically in two ways. First, homogeneity analyses are used to check whether the grouping of effect sizes from different studies shows more variation than would be expected from sampling error alone, indicating that the effect sizes may not be comparable. A second approach to this problem is to break down the overall mean effect size into mean effect sizes for different types of tasks. For these mean effect sizes per type of task to be meaningful, we (subjectively) set a minimum of five eligible studies per sub-task analysis. This led to the exclusion of two studies using tasks assessing the attribution of mental states to abstract shapes rather than humans (Blakemore et al, 2003; Russell et al, 2006), and three studies in which the `eyes' task was used (Kington et al, 2000; Russell et al, 2000; Kelemen et al, 2005).
Schizophrenia subgrouping
Ever since Frith's first proposal
(Frith, 1992), the association
between mentalising and the core symptoms of schizophrenia has been an
important focus of research interest. Schizophrenia is a heterogeneous
disorder and various subgrouping methods have been used, based on different
theories regarding the relationship between mentalising and
symptomatology.
In earlier studies, Frith and colleagues divided their schizophrenia samples into six symptom subgroups (Corcoran et al, 1995). In their later studies, the number of subgroups was reduced to four, categorised as follows:
The first group was predicted to be the most impaired, because of these patients' incapacity to represent the mental states of others as well as themselves. Paranoid patients would perform poorly because of their difficulties in monitoring other people's intentions. Patients whose symptoms were in remission and patients with passivity symptoms were predicted to have normal mentalizing abilities. These hypotheses were largely confirmed and have repeatedly been replicated (Frith & Corcoran, 1996; Corcoran et al, 1997; Pickup & Frith, 2001).
Sarfati and colleagues (Sarfati et al, 1997a,b, 1999; Sarfati & Hardy-Baylé, 1999) and Zalla et al (2006) suggested that impairment of theory of mind is related to thought disorder, reflecting an executive functioning deficit. Thus, their samples were divided into those with and those without thought disorder. In all of their studies thought-disordered participants performed significantly more poorly than healthy controls. However, in two of the studies the non-disorganised participants also showed poor performance (Sarfati et al, 1997b; Zalla et al, 2006).
Three research groups studied the relationship between mentalising and paranoid delusions (Randall et al, 2003; Craig et al, 2004; Harrington et al, 2005b). In all three studies patients with paranoid delusions showed impairment of theory of mind relative to the normal control group. However, in the study by Randall et al (2003), theory of mind performances of the paranoid and non-paranoid subgroups did not differ significantly from each other. Lastly, Herold et al (2002) investigated whether the deficit in theory of mind was state- or trait-dependent and therefore assessed patients whose schizophrenia was in remission. Results showed that theory of mind impairment was still present in the remission phase of the illness.
Moderator variables
Published research suggests a number of variables that may affect
mentalising performance and thus influence effect size. Hence, we aimed to
code these variables in order to evaluate their influence on the effect size.
Potential moderator variables at individual patient level are age, gender,
medication, IQ, disease status (acute, chronic or in remission), severity of
psychopathology, and symptoms. To analyse the effect of specific clusters of
symptoms on mentalising impairment, the symptom subgroups used by different
research groups were divided into four categories:
The disorganised subgroup was composed of the behavioural symptoms subgroup of the studies by Frith and colleagues (Corcoran et al, 1995, 1997; Pickup & Frith, 2001) and the disorganised subgroups of the Sarfati, Mazza and Zalla studies (Sarfati et al, 1997a,b, 1999; Sarfati & Hardy-Baylé, 1999; Mazza et al, 2001; Zalla et al, 2006). The non-disorganised patients of the Sarfati and Zalla studies were combined into the second subgroup (Sarfati et al, 1997a,b, 1999; Sarfati & Hardy-Baylé, 1999; Zalla et al, 2006). For the paranoid subgroup the results of the studies focusing on paranoid schizophrenia (Randall et al, 2003; Craig et al, 2004; Harrington et al, 2005b) were combined with the results for the paranoid subgroups of the studies by Frith and colleagues (Corcoran et al, 1995, 1997; Pickup & Frith, 2001). The remitted disease subgroup comprised the patients in remission in the studies by Herold et al (2002), Randall et al (2003) and Frith and colleagues (Corcoran et al, 1995; Corcoran et al, 1997; Pickup & Frith, 2001). The passivity subgroup of Frith and colleagues was not coded, because results for that subgroup were reported only in two studies.
Potential moderators at study level are the matching of patients and controls on group characteristics (e.g. mean age, mean IQ, gender distribution), type of mentalising task used, and whether the task is administered verbally or non-verbally. Four types of theory of mind tasks were distinguished: first-order false belief/deception; second-order false belief/deception; intention inferencing; and comprehension of indirect speech. Some tasks did not fit in any of these categories, for example the false belief/deception tasks for which the orders were unknown or mixed.
Within the different task paradigms there is also variation in whether tasks are presented in a verbal or non-verbal form. It has been suggested that verbalisation may be impoverished in schizophrenia and may constitute an experimental bias in favour of a theory of mind deficit in people with schizophrenia (e.g. Sarfati et al, 1999). In a separate coding, tasks were classified as verbal or non-verbal.
Coding
Each study was coded independently by two authors (M.S. and E.V.). In case
of discrepancies, consensus was reached in conference with the whole research
group. When results were reported in graphical form only an email was sent to
the author with a request for the exact numerical results.
Data collection and analysis
For each study an unbiased standardised mean difference (d), was
calculated using reported means and standard deviations. This effect size
statistic is computed as the difference between the mean of the schizophrenia
group and the mean of the control group, divided by the pooled standard
deviation. Hedges' formula was applied to correct for upwardly biased
estimation of the effect size in small samples
(Lipsey & Wilson,
2001).
When means and standard deviations were not available, d was
calculated from the reported t or F values. In cases where
the only reported outcome variable was the proportion of participants with a
good (or poor) performance, d was estimated using the probit
transformation method (Lipsey &
Wilson, 2001). A sensitivity analysis was performed to check
whether there was any significant effect of using probit-transformed effect
sizes on the overall effect size. In studies in which data were reported for
(symptom) subgroups only, data were first pooled and then compared as one
group with the control group. In addition, the effect sizes of symptom
subgroups were calculated for subsequent analyses. Several studies used more
than one (sub)task to assess theory of mind, and therefore had more than one
effect size; in these cases a pooled effect size was computed. However, if the
authors had included a composite score, the effect size of this score was
calculated. Again, effect sizes for different task types were calculated for
subsequent analyses. In addition to the individual effect sizes and 95%
confidence intervals, P values were calculated for each study using
two-tailed independent t-tests and
2-tests.
The mean effect size across studies was calculated by weighting each effect size by the inverse of its sampling variance. A confidence interval and z-value were calculated to examine the statistical significance of the effect. To test whether the individual effect sizes are good estimators of the population effect size, the homogeneity statistic Q was calculated (Lipsey & Wilson, 2001). Because sample sizes are small in the subgroup and task type analyses (see below), a random effects model was fitted to the data (Lipsey & Wilson, 2001). To examine publication bias, a fail-safe number was computed using Orwin's formula (Lipsey & Wilson, 2001). This indicates the number of studies with null effects that have to reside in file drawers to reduce the mean effect size to a negligible level (which we set at 0.2). Weighted regression analysis was performed using the statistical package Meta-Stat (Rudner et al, 2002) to evaluate whether group differences in IQ, gender and age had an impact on effect size. Other variables with a potential influence on effect size, such as patient status, medication use and severity of psychopathology, could not be analysed because of the small number of studies reporting results for these parameters. Separate analyses were performed to analyse whether mentalising impairment is different for different symptom subgroups or for different types of mentalising tasks.
|
|
|---|
|
View this table: [in a new window] |
Table 1 Summary of studies included in the meta-analysis
|
Analysis of the total sample
Figure 1 shows the 29
individual effect sizes with their 95% confidence intervals. None of the
confidence intervals includes the value zero, indicating a statistically
significant effect for each study. The weighted mean effect size of the
combined sample is 1.255 (95% CI 1.441 to 1.069) which is
also statistically significant (z=13.25, P<0.0001).
Homogeneity analysis showed that there was homogeneity among studies
(Q=29.13, d.f.=28, P<0.41), and weighted regression
analysis did not show any relationship between effect size and difference
between patient and control groups in IQ (P=0.193), proportion of
males (P=0.115) and age (P=0.147). The fail-safe number was
153, which indicates that 153 unpublished studies are required to reduce the
effect size of the combined findings to a negligible level.
![]() View larger version (19K): [in a new window] [as a PowerPoint slide] |
Fig. 1 Individual and mean effect sizes (d) and 95% confidence intervals
of mentalising deficits in schizophrenia.
|
![]() View larger version (16K): [in a new window] [as a PowerPoint slide] |
Fig. 2 Mean effect sizes (d) and 95% confidence intervals of mentalising
deficits in symptom subgroups of schizophrenia: group 1, with disorganisation
(n=9); group 2, without disorganisation (n=5); group 3,
paranoid (n=6); group 4, remission (n=5).
|
Analyses of the types of mentalising tasks
The mean effect sizes and confidence intervals of the four theory of mind
task categories are shown in Fig.
3. The mean effect sizes of the first-order tasks
(d=1.193, 95% CI 1.666 to 0.720,
P<0.01) and the second-order tasks (d=1.443, 95%
CI 1.867 to 1.019, P<0.01) have homogeneity
statistics indicating heterogeneity among the effect sizes: Q=97.691,
d.f.=12 (P<0.01) and Q=17.875, d.f.=6
(P<0.01) respectively. In contrast, the mean effect sizes within
both the indirect speech tasks (d=<1.040, 95% CI 1.301 to
0.779, P<0.01) and the intention-inferencing tasks
(d=0.959, 95% CI 1.228 to 0.690,
P<0.01) are both homogeneous. The difference between the mean
effect sizes for different subtasks could not be analysed statistically,
because not all effect sizes were statistically independent since in one study
different types of tasks might have been used.
![]() View larger version (14K): [in a new window] [as a PowerPoint slide] |
Fig. 3 Mean effect sizes (d) and 95% confidence intervals of mentalising
deficits for different types of mentalising tasks: 1, first-order false belief
and deception tasks (n=13); 2, second-order false belief and
deception tasks (n=7); 3, tasks assessing the comprehension of
indirect speech (n=8); 4, intention-inferencing tasks
(n=7).
|
|
|
|---|
The moderator variables IQ, gender and age did not significantly affect mean effect size. Thus, the impairment in theory of mind is robust and is not readily moderated by variables that may seem relevant. However, the effect of other potentially important moderator variables such as medication use and duration and severity of illness could not be analysed owing to a lack of information on these characteristics in many studies.
Participants with schizophrenia who had signs and symptoms of disorganisation were found to be significantly more impaired in terms of theory of mind than those in the other symptom subgroups. However, these results may also be explained by the composition of the disorganised symptom subgroup. The behavioural subgroup of the studies by Frith and colleagues was ranked highest in their hierarchical model. Thus, individuals in this group might also have had symptoms of the paranoid and/or passivity subgroup. This brings the risk that poorer performance in this group may be explained by having more severe and complex symptoms (Harrington et al, 2005a). Similarly, in two of the four studies by Sarfati and colleagues the disorganised subgroup had more general psychopathology, which might explain their poorer theory of mind performance (Sarfati & Hardy-Baylé, 1999; Sarfati et al, 1999).
The mean effect size (d=0.692) of mentalising impairment in patients in remission was smaller than in the other symptom subgroups, but is still considered to be medium to large (Cohen, 1988). Moreover, this effect did not differ significantly from the effect sizes of the disorganised and paranoid subgroups.
Unexpectedly and despite apparent differences in type and difficulty of the theory of mind tasks the mean effect sizes for different task types were found to be similar. An explanation might be that our method of grouping studies by task types was not correct. This is supported by the finding that two of the four task type analyses showed heterogeneity among effect sizes. However, since there is a lack of research on the psychometric properties of the tasks that were used, such as construct and concurrent validity, it is not yet possible to categorise these tasks objectively.
There was also no difference between the mean effect sizes of verbal and non-verbal tasks, which is consistent with the findings of Sarfati and colleagues (Sarfati et al, 1999, 2000). Thus, impairment of theory of mind does not to appear to be affected by verbalisation deficits that have been reported in people with schizophrenia.
Mentalising in schizophrenia: generalised v. specific impairment
As shown by Heinrichs & Zakzanis
(1998), people with
schizophrenia show generalised neurocognitive impairment. On their list of 22
mean effect sizes of common neurocognitive tests, the effect size of
mentalising impairment would be ranked fourth. An interesting question is
whether poor mentalising performance in schizophrenia interacts with or is
influenced by general cognitive impairment. This problem is acknowledged by
some authors, who corrected for general cognitive abilities by matching groups
on IQ, covarying out cognitive variables (e.g. attention, executive
functioning, memory, general picture sequencing abilities) or excluding
participants from statistical analyses if they answered reality questions
about the theory of mind stories incorrectly. In their reviews, Brüne ne
(2005a: p. 25,
Table 1) and Harrington et
al (2005a: pp.
252267, Table 1)
discussed the empirical evidence as to whether the mentalising deficits in
schizophrenia are specific or the consequence of general cognitive impairment.
In both reviews it was concluded that the evidence speaks in favour of the
notion that there is a specific theory of mind deficit in schizophrenia. As
with many neurocognitive tests, theory of mind tasks probably measure several
component processes at the same time. For example, tasks in which the
comprehension of indirect speech is assessed may require not only mentalising
abilities but also basic language comprehension and expressive language
skills. Possibly, general cognitive abilities represent a necessary but not
sufficient condition for adequate mentalising, which is known as the `building
block' view of social cognition (see Penn
et al, 1997).
Mentalising in schizophrenia: state or trait dependency
In his cognitive model of the relationship between meta-representation and
the signs and symptoms of schizophrenia, Frith assumed that in people with
this disorder, the initial development of mentalising abilities is relatively
normal and that these abilities become impaired as the illness develops
(Frith, 1992). In the
subsequent studies by him and his colleagues, it was predicted and found that
patients who were in remission (i.e. symptom-free) were unimpaired compared
with normal controls (e.g. Corcoran et al,
1995,
1997;
Frith & Corcoran, 1996;
Pickup & Frith, 2001). In
contrast, our meta-analysis has shown that patients have significant
impairment during remission, which is consistent with the findings of Herold
et al (2002). These
findings support the notion that mentalising is not just a consequence of the
acute phase of the disorder but may be trait-dependent. It cannot be excluded
that the criteria for remission (e.g. partial or full remission) used by
Herold et al (2002)
and by Frith and colleagues are different. Other factors such as
(prophylactic) treatment may also explain the divergent findings. However,
more support for the trait argument comes from studies on mentalising in
populations at elevated risk of developing a psychotic illness.
In general, people at genetic risk of schizophrenia show reduced performance on the more common types of theory of mind tasks (Wykes et al, 2001; Irani et al, 2006; Marjoram et al, 2006), but not on the `eyes' test (Kelemen et al, 2004; Irani et al, 2006). In the study by Schiffman et al (2004), genetic high-risk children who would later develop schizophrenia-spectrum disorders had lower scores on a role-taking task, which the authors considered assessed a facet of theory of mind. An association between theory of mind performance and subclinical schizotypal traits has also been found (Langdon & Coltheart, 1999, 2004; Irani et al, 2006; Meyer & Shean, 2006). Pickup (2006) showed that schizotypal traits analogous to positive symptoms of schizophrenia predicted poorer mentalising performance, whereas no association was found between poorer theory of mind and schizotypal traits analogous to the `behavioural signs' of schizophrenia. Platek et al (2003) suggested that contagious yawning is part of a more general phenomenon known as mental state attribution. Consistent with this hypothesis, susceptibility to contagious yawning was positively related to performance on (other) mentalising tasks, and negatively related to schizotypal personality traits. Only in the study by Jahshan & Sergi (2007) was there no difference between people with high schizotypy and those with low schizotypy regarding theory of mind performance. There is thus considerable evidence that mentalising impairment is a susceptibility indicator for schizophrenia and hence may be trait-dependent.
Limitations
The first limitation, to which we have already alluded, is that studies
were excluded in which less common types of theory of mind tasks were used.
Because there is no information on the psychometric properties of the many
different tasks, this is somewhat arbitrary. In addition, the categorisation
of task type is not supported by psychometric evidence. Second, the method of
categorising symptom subgroups employed in this meta-analysis should be
considered tentative. The main problem with our approach is that there is
overlap between symptom clusters; for example, the subgrouping method used by
Frith and colleagues is hierarchical, with the behavioural subgroup being the
highest category. This means that patients in that subgroup could also report
paranoid symptoms, but those in the paranoid subgroup could not report
behavioural symptoms. As another example, participants categorised as paranoid
in the study by Harrington et al
(2005b) could also
have formal thought disorder (which was indeed the case). However, in spite of
this limitation, we believe that the results of the subgroup analyses in this
meta-analysis are valuable. This is statistically supported by the homogeneity
analyses, which show that the clustering of symptom subgroups did not result
in more variation than would be expected from sampling error alone and that it
is plausible that the studies within the subgroup analyses are comparable.
Recommendations for future research
The results and limitations of this meta-analysis lead to some
recommendations for future research. First, research focusing on the
mentalising process itself is necessary, addressing questions on what
components it comprises and on how to operationalise them. As has already been
pointed out by Harrington et al
(2005a), it is also
important to establish the psychometric properties of theory of mind tasks.
Second, the finding that the deficit in theory of mind in schizophrenia is
perhaps trait-dependent rather than state-dependent implies that the deficit
may also be present before illness onset. Therefore, there may be a role of
mentalising impairment in the early detection and prediction of schizophrenia,
requiring a longitudinal study examining theory of mind abilities in people at
risk of developing schizophrenia.
Third, the finding that theory of mind impairment may be trait-dependent also brings to mind a comparison with autism-spectrum disorders. An impaired ability to understand mental states has been described as one of the core symptoms of such disorders (Yirmiya et al, 1998). However, although the risk of psychotic disorder is elevated in individuals with autism-spectrum disorder (Stahlberg et al, 2004), most of them will not develop a psychotic disorder. Future research should focus on what the commonalities and differences are with regard to theory of mind in these disorders. Abu-Akel & Bailey (2000) for example suggested that there might be different forms of impairment of theory of mind. They argue that, unlike people with autism-spectrum disorders, people with schizophrenia do not lack an understanding that others have mental states; instead, they may overattribute knowledge to others or apply their knowledge of mental states in an incorrect or biased way. Thus, an interesting research topic would be a comparison of the mentalising abilities of groups of people with these two disorders.
Lastly, social impairment is one of the most disabling clinical features of schizophrenia and it is well known that it is often present before illness onset (e.g. Niemi et al, 2003). Since theory of mind impairment appears to be trait- rather than state-dependent in schizophrenia, this deficit may have a role in the development of social impairment. However, evidence of a relationship between theory of mind performance and social functioning is lacking and should be an aim of future research.
|
|
|---|
|
|
|---|
Related articles in BJP:
This article has been cited by other articles:
![]() |
H. Walter, A. Ciaramidaro, M. Adenzato, N. Vasic, R. B. Ardito, S. Erk, and B. G. Bara Dysfunction of the social brain in schizophrenia is modulated by intention type: An fMRI study Soc Cogn Affect Neurosci, June 1, 2009; 4(2): 166 - 176. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. Tyrer From the Editor's desk The British Journal of Psychiatry, November 1, 2008; 193(5): 434 - 434. [Full Text] [PDF] |
||||
![]() |
B. King-Casas, C. Sharp, L. Lomax-Bream, T. Lohrenz, P. Fonagy, and P. R. Montague The Rupture and Repair of Cooperation in Borderline Personality Disorder Science, August 8, 2008; 321(5890): 806 - 810. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. K. Wynn, J. Lee, W. P. Horan, and M. F. Green Using Event Related Potentials to Explore Stages of Facial Affect Recognition Deficits in Schizophrenia Schizophr Bull, July 1, 2008; 34(4): 679 - 687. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Park, N. Matthews, and C. Gibson Imitation, Simulation, and Schizophrenia Schizophr Bull, July 1, 2008; 34(4): 698 - 707. [Abstract] [Full Text] [PDF] |
||||
![]() |
N. McGlade, C. Behan, J. Hayden, T. O'Donoghue, R. Peel, F. Haq, M. Gill, A. Corvin, E. O'Callaghan, and G. Donohoe Mental state decoding v. mental state reasoning as a mediator between cognitive and social function in psychosis The British Journal of Psychiatry, July 1, 2008; 193(1): 77 - 78. [Abstract] [Full Text] [PDF] |
||||
![]() |
E. Pousa, A. I. Ruiz, and A. S. David Mentalising impairment as a trait marker of schizophrenia? The British Journal of Psychiatry, April 1, 2008; 192(4): 312 - 312. [Full Text] [PDF] |
||||
Read all eLetters
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||