© 2006 The Royal College of Psychiatrists
Influence of sub-syndromal symptoms after remission from manic or mixed episodes
Lilly Research Laboratories, Indianapolis, Indiana and Department of Psychiatry, McLean Hospital, Harvard Medical School, Belmont, Massachusetts
Department of Psychiatry, University of Texas Health Sciences Center, San Antonio, Texas
Department of Psychiatry, Case Western Reserve University, Cleveland, Ohio
Lilly Research Laboratories, Indianapolis, Indiana
Department of Psychiatry, Harvard Medical School and Massachusetts General Hospital, Boston, Massachusetts, USA
Centro Lucio Bini, Rome, Italy
University of British Columbia, Vancouver, Canada
Department of Psychiatry, University of Munich, Germany
Correspondence: Dr MauricioTohen, Lilly Research Laboratories, Indianapolis, IN 46285, USA. Tel: +1(317) 277 9585; fax: +1(317) 276 7845; email: m.tohen{at}lilly.com
Declaration of interest Sponsorship from Lilly Research Laboratories.
 TOP ABSTRACT INTRODUCTIONMETHODRESULTSDISCUSSIONREFERENCES |
|---|
Background Sub-syndromal symptomsin bipolar disorder impair functioning and diminish quality of life.
Aims To examine factors associated with time spent with sub-syndromal symptoms and to characterise how these symptoms influence outcomes.
Method In a double-blind randomised maintenance trial, patients received either olanzapine or lithium monotherapy for 1 year. Stepwise logistic regression models were used to identify factors that were significant predictors of percentage time spent with sub-syndromal symptoms. The presence of sub-syndromal symptoms during the first 8 weeks was examined as a predictor of subsequent relapse.
Results Presence of sub-syndromal depressive symptoms during the first 8 weeks significantly increased the likelihood of depressive relapse (relative risk 4.67, P<0.001). Patients with psychotic features and those with a greater number of previous depressive episodes were more likely to experience sub-syndromal depressive symptoms (RR=2.51, P<0.001 and RR=2.35, P=0.03 respectively).
Conclusions These findings help to identify patients at increased riskof affective relapse and suggestthat appropriate therapeutic interventions should be considered even when syndromal-level symptoms are absent.
|
TOPABSTRACTINTRODUCTIONMETHODRESULTSDISCUSSIONREFERENCES |
|---|
Recent attention has focused on the need to develop therapeutic strategies that allow patients with bipolar affective disorder to return to a premorbid level of function. Effective therapies are currently available for the treatment of acute manic and depressive episodes, and for prolonging time in remission. However, patients considered to have responded to treatment may nevertheless continue to experience subsyndromal symptoms that impair functioning and diminish quality of life. This disparity between symptom amelioration and functional outcomes among patients with bipolar disorder has been described by Tohen et al (1990a) and Chengappa et al (2005). In a study of first-episode disorder, Tohen et al (2000) observed that only 38% of patients achieved functional recovery within 2 years of treatment for an acute manic episode. One factor that may contribute to the difficulty in reaching full functional recovery is the presence of subsyndromal symptoms that either individually or as an aggregate are not sufficiently severe to constitute a major mood episode, but interfere with functioning.
A recent reassessment of bipolar-spectrum disorders including patients with sub-syndromal symptoms revealed at least a five-fold greater prevalence than found with traditionally defined syndromal diagnoses (Judd & Akiskal, 2003). The persistence of sub-syndromal symptoms in patients with bipolar disorder has been shown to contribute substantially to functional impairment (Gitlin et al, 1995; Altshuler et al, 2002). Furthermore, subsyndromal symptoms are associated with an increased risk of relapse (Goodnick et al, 1987; Tohen et al, 1990a; Keller et al, 1992). Keller et al (1992) demonstrated that patients prescribed lithium who achieved standard serum levels of the drug were both less likely to develop sub-syndromal symptoms and less likely to experience relapse relative to those who achieved low-range levels, which highlights the role of adequate therapeutic treatment in the management of these symptoms.
A critical part of developing effective treatment strategies for the management of both sub-syndromal and syndromal symptoms requires a greater understanding of the factors involved in their development. The goal of our post hoc analyses was to identify factors that are associated with a greater percentage of time with sub-syndromal symptoms, and to characterise how these symptoms influence outcomes in a randomised double-blind clinical trial of relapse prevention comparing olanzapine and lithium in patients with bipolar I disorder.
|
TOPABSTRACTINTRODUCTIONMETHODRESULTSDISCUSSIONREFERENCES |
|---|
The details of this randomised double-blind controlled trial have been described previously (Tohen et al, 2005) and are summarised briefly here. Participants were at least 18 years old and met DSM–IV (American Psychiatric Association, 1994) criteria for an index manic or mixed bipolar I episode based on the Structured Clinical Interview for DSM–IV (First et al, 1995). Prerequisites for study entry were a baseline total score of 20 or more on the Young Mania Rating Scale (YMRS; Young et al, 1978) and a history of at least two manic or mixed episodes within the previous 6 years.
The trial consisted of four study periods: screening (2–7 days); open-label co-therapy (6–12 weeks); double-blind taper (4 weeks); and double-blind monotherapy (48 weeks). The starting daily dosages for open-label co-therapy were olanzapine 15 mg and lithium 600 mg. Subsequent dosages of olanzapine could range from 5 mg to 20 mg per day. Investigators were required to optimise lithium dosage and reach a target serum level of 0.6–1.2 mmol/l by week 4 during this period.
Patients who met symptomatic remission criteria for bipolar disorder – a total YMRS score of 12 or below and a total score of 8 or below on the 21-item Hamilton Rating Scale for Depression (HRSD; Hamilton, 1959) – during the second study period (co-therapy) were randomly assigned to treatment with either olanzapine or lithium monotherapy. During the third study period, patients remained on their current dosage of randomised treatment and the dosage of the discontinued drug was tapered over 4 weeks. During the final study period, lithium levels were monitored and the dosage adjusted to maintain serum levels in the therapeutic range 0.6–1.2 mmol/l. All patients randomly assigned to olanzapine also had blood drawn to maintain the study masking. For every outlier report generated for a patient in the lithium group, a sham lithium outlier report was sent to a patient in the olanzapine group. Thus, reports to investigative sites indicating that the lithium dosage should be adjusted did not unmask the randomised assignment.
Illness severity was assessed using the YMRS and HRSD. The categorical definitions of euthymia, sub-syndromal symptoms and relapse (Table 1) are based on previous reports (Yatham et al, 2004; Gopal et al, 2005; Tohen et al, 2005). Relapse and sub-syndromal symptoms were classified globally as bipolar (any mood symptom) and by their respective manic and depressive poles.
|
View this table: [in a new window] | Table 1 Categorical definitions of euthymia, sub-syndromal status and relapse |
Statistical methods
Stepwise logistic regression models were used to identify factors that were
significant predictors of the proportion of time spent with sub-syndromal
symptoms. The response, percentage time spent with subsyndromal symptoms, was
broken into five categories by quartiles: 0%, 0–<25%,
25%–<50%, 50%–<75% and 75% or more. Linear logistic
regression models for these ordinal response data were fitted by the method of
maximum likelihood using SAS (version 8.2) PROC LOGISTIC. Categorical
explanatory variables included therapy, gender, index episode type (mania or
mixed), presence of psychotic features, rapid-cycling rapid-cycling status,
proneness to depression or mania (defined by the predominant episode type in
previous episodes), number of previous bipolar episodes (0–5, 6–9,
10 or more), number of previous manic episodes (0–2, 3–5, 6 or
more) and number of previous depressive episodes (0–1, 2–3, 4 or
more). Dimensional explanatory variables included age, onset age, duration of
illness, baseline YMRS total score, and baseline HRSD total score. The
variable therapy was included in all models regardless of statistical
significance. The other explanatory variables were entered into the model if
they were significant at the 
=0.05 level and could be removed in a
stepwise manner if the least significant effect in the model at a particular
step did not meet this level of significance for staying in the model. Odds
ratio estimates and their corresponding 95% confidence intervals were
calculated for explanatory variables in the final model.
The presence of sub-syndromal symptoms during the first 8 weeks was
examined as a predictor of subsequent relapse. Separate analyses were made for
each type of sub-syndromal status (any episode, depressive, manic) and each
type of relapse (any episode, depressive, manic). Patients who completed at
least 8 weeks of therapy without relapse (n=340) were included in the
analysis. Additional analyses evaluated the potential impact of the presence
of residual symptoms at the outset of the double-blind treatment phase and of
the absence of symptoms at the initiation of the double-blind period followed
by the emergence of sub-syndromal symptoms during the first 8 weeks on the
subsequent risk of relapse: patients were stratified according to the presence
(YMRS score 9–14) or absence (YMRS score 
8) of residual symptoms at
the outset of the double-blind period. Fisher’s exact test was used to
test proportions, and relapse incidence rates were characterised by an
estimate of the relative risk with 95% confidence limits; estimates were
constructed with patients having no time with sub-syndromal symptoms during
the first 8 weeks as the referent.
|
TOPABSTRACTINTRODUCTIONMETHODRESULTSDISCUSSIONREFERENCES |
|---|
Our analyses included 424 patients with bipolar I disorder. Their baseline demographic and illness characteristics are presented in Table 2. No statistically significant difference was observed between the olanzapine and lithium treatment groups on these baseline measures.
|
View this table: [in a new window] | Table 2 Demographic and illness characteristics of the sample |
The percentages of patients with subsyndromal symptoms at any time during the 48-week study are shown in Table 3. Presented in Table 4 are the percentages of time spent with sub-syndromal symptoms categorised by quartiles. There was no statistically significant difference between therapies with respect to percentage of patients with sub-syndromal symptoms at any time, or the percentage of time with sub-syndromal bipolar symptoms overall or in the individual poles (depression or mania). The mean percentages of time spent with any mood symptom or with manic or depressive sub-syndromal symptoms among patients who experienced them were 27.4% (s.d.=25.7), 26.8% (s.d.=28.7), and 24.4% (s.d.=18.1) respectively (median times 18.4%, 15.4% and 21.5% respectively).
|
View this table: [in a new window] | Table 3 Participants with sub-syndromal symptoms at any time during the 48-week study |
|
View this table: [in a new window] | Table 4 Percentage of time spent with sub-syndromal symptoms |
Among patients who completed the first 8 weeks of the relapse prevention phase without a major affective episode (n=340), the subsequent rate of relapse for those with bipolar sub-syndromal symptoms was 36.8% (32 of 87) and 29.6% (75 of 253) for those without. Regardless of therapy, the presence of sub-syndromal bipolar symptoms significantly increased the likelihood of relapse into the depressive pole (17 of 87 patients with sub-syndromal bipolar symptoms v. 26 of 253 patients without such symptoms; relative risk 1.9, 95% CI 1.09–3.33, P=0.038). The presence of sub-syndromal depressive (but not manic) symptoms also increased the likelihood of bipolar relapse (15 of 26 patients with symptoms v. 92 of 314 patients without; RR=1.97, 95% CI 1.36–2.85, P=0.004) and relapse into the depressive (but not manic) pole (12 of 26 patients with symptoms v. 31 of 314 patients without; RR=4.67, 95% CI 2.74–7.97, P<0.001). Patients who started the relapse prevention phase without subsyndromal bipolar symptoms but developed them during the first 8 weeks were significantly more likely to experience bipolar relapse relative to those without sub-syndromal symptoms during the same period (20 of 39 patients v. 75 of 253 patients; RR=1.73, 95% CI 1.21–2.48, P=0.01). This was especially true of patients with sub-syndromal depressive symptoms and subsequent relapse into the depressive pole (12 of 25 patients v. 31 of 314 patients; RR=4.86, 95% CI 2.87–8.24, P<0.0001). In contrast, patients who had residual sub-syndromal symptoms (any mood) at the outset of the double-blind phase and continued to experience symptoms during the first 8 weeks were not more likely to relapse than those without sub-syndromal symptoms during this period (12 of 48 patients v. 75 of 253 patients; RR=0.84, 95% CI 0.50–1.43, P=0.60).
Of the individual factors assessed, presence of psychotic features and a greater number of previous depressive episodes were associated with increased time with depressive sub-syndromal symptoms. Patients with psychotic features were 2.51 (95% CI 1.47–4.30; P<0.001) times more likely to experience sub-syndromal depressive symptoms. Patients with two or three previous depressive episodes were 1.96 (95% CI 1.04–3.71) times more likely to experience sub-syndromal depressive symptoms relative to those with no or one previous episode, and those with four or more were 2.35 (95% CI 1.21–4.54) times more likely to do so than those with no or one episode (P=0.03).
|
TOPABSTRACTINTRODUCTIONMETHODRESULTSDISCUSSIONREFERENCES |
|---|
Our analyses identified illness characteristics that were associated with a greater amount of time spent experiencing subsyndromal symptoms in patients with bipolar I disorder, and examined the impact of these symptoms on outcomes. There was no statistically significant difference between the olanzapine and lithium treatment groups with regard to the percentage of patients who experienced sub-syndromal symptoms or in the percentage of time spent with symptoms. Approximately 38% of patients experienced symptoms that fell within the sub-syndromal range of severity, as defined by rating scales, at any time during the maintenance phase of this study. Among these patients, just over a quarter (27%) of the time was spent with sub-syndromal symptoms. These results are consistent with previous reports from non-controlled studies documenting the prevalence of sub-syndromal symptoms in patients with bipolar I disorder (Keitner et al, 1996; Judd et al, 2002; Post et al, 2003). Our findings further extend the view that sub-syndromal symptoms are common and pervasive in bipolar I disorder, even in a population of patients who achieved clinical response from an acute manic or mixed episode and continued to receive treatment for relapse prevention.
Meaningful comparisons between these findings and those of previous reports with respect to time with sub-syndromal symptoms are difficult to make because of differences in defining criteria, methods of assessment and study design. Notably, for participants in our study who experienced relapse into an affective episode during the study period, only data prior to the event were included in the analyses. It should also be noted that all patients entered this study with an index manic or mixed episode, which may account in part for differences in the distribution of time with symptoms in the two poles relative to studies that involved patients with either manic/mixed or depressive episodes. Indeed, previous studies have reported that patients spend substantially more time with depressive symptoms than with manic symptoms (Judd et al, 2002; Post et al, 2003; Joffe et al, 2004), whereas the distribution of time with symptoms in our study was roughly equal for the manic (26.8%) and depressive (24.4%) poles. Since the polarity of index mood episode is a predictor of the polarity of subsequent relapse (Tohen et al, 2003), it is possible that the distribution of sub-syndromal symptoms is similarly dependent on the polarity of the index episode.
Predictors of sub-syndromal symptoms
Of the clinical variables analysed, the presence of psychotic features and
the number of previous depressive episodes were associated with increased time
spent with sub-syndromal symptoms. Patients who entered the study with
psychotic features were more likely to experience a greater percentage of time
with subsyndromal depressive symptoms than those without such features. This
finding is interesting in light of recent reports questioning the prognostic
value of psychotic features in bipolar disorder with respect to illness
severity and treatment response (Keck
et al, 2003). In a study by Swann et al
(2004) the presence of
psychotic features was associated with greater overall functional impairment,
but was not correlated with higher baseline mania scores or altered treatment
response. On the other hand, a study by Tohen et al
(1990b) identified
the presence of psychotic features during the index episode as a predictor of
shorter time in remission.
A greater number of previous depressive episodes was also associated with a greater percentage of time spent with subsyndromal depressive symptoms, which is in accordance with a previous report by Post et al (2003). These findings also extend previous reports that a greater number of previous affective episodes increases the risk of subsequent relapse (Kessing et al, 2004).
Predictors of relapse
In our study the presence of sub-syndromal symptoms during the first 8
weeks of the relapse prevention phase was associated with a significantly
greater likelihood of subsequent relapse, particularly into the depressive
pole. This finding agrees, in part, with previous studies that have reported
an increased risk of relapse associated with sub-syndromal symptoms
(Goodnick et al, 1987;
Tohen et al,
1990a; Keller et
al, 1992). However, the presence of depressive sub-syndromal
symptoms was predictive of depressive relapse, whereas there was no
corresponding relationship between sub-syndromal manic symptoms and manic
relapse. In contrast, in a study of relapse prevention with lithium, Keller
et al (1992) reported
in a non-controlled study a stronger association between manic subsyndromal
symptoms and manic relapse relative to the depressive polarity. It is not
clear what factors account for these discrepant results; however, it is
possible that the use of olanzapine and lithium in combination to treat the
acute episode, and monotherapy during relapse prevention, might have
contributed to our results. Further analyses of sub-syndromal symptoms during
the first 8 weeks of the monotherapy phase that differentiated residual
subsyndromal symptoms (i.e. symptoms from the index episode that had not
resolved completely) from newly emerged subsyndromal symptoms yielded
intriguing results: the emergence of sub-syndromal symptoms (in particular
sub-syndromal depressive symptoms) was associated with significantly greater
risk of subsequent relapse into the depressive pole, whereas the presence of
residual symptoms was not. This finding suggests that the longitudinal
assessment of symptom severity, as opposed to just cross-sectional assessment,
might better determine the risk of subsequent relapse.
Limitations
There are several limitations to these analyses that warrant discussion.
Patients in this study were required to achieve remission from an acute manic
or mixed episode to be included in the relapse prevention phase of the trial;
thus, this population consisted of patients who responded to combined
olanzapine and lithium treatment and who might not be representative of the
general population of people with bipolar I disorder. A related limitation is
that our results may not be generalised to patients who are unable or refuse
to participate in clinical trials. It should be noted, however, that in
contrast to naturalistic studies, the analyses of data from our randomised
clinical trial were adjusted for pharmacological treatment. Another limitation
is the relatively short follow-up period of only 1 year, which may not
characterise the full longitudinal course of bipolar disorder.
Our findings provide prognostic value in terms of identifying patients at increased risk of affective relapse. Given that the presence of sub-syndromal symptoms is associated with significant functional impairment (Tohen et al, 1990a; Altshuler et al, 2002) and incurs large social and economic health costs (Bauer et al, 2001; MacQueen et al, 2003), appropriate pharmacological and non-pharmacological therapeutic interventions should be considered even in the absence of symptoms at the syndromal level.
|
TOPABSTRACTINTRODUCTIONMETHODRESULTSDISCUSSIONREFERENCES |
|---|
- Altshuler, L. L., Gitlin, M. J., Mintz, J., et al (2002) Subsyndromal depression is associated with functional impairment in patients with bipolar disorder. Journal of Clinical Psychiatry, 63, 807 –81.
- American Psychiatric Association (1994) Diagnostic and Statistical Manual of Mental Disorders (4th edn) (DSM–IV). Washington, DC: APA.
- Bauer, M. S., Kirk, G. F., Gavin, C., et al (2001) Determinants of functional outcome and healthcare costs in bipolar disorder: a high-intensity follow-up study. Journal of Affective Disorders, 65, 231 –241.[CrossRef][Medline]
- Chengappa, K. N., Hennen, J., Baldessarini, R. J., et al (2005) Recovery and functional outcomes following olanzapine treatment for bipolar I mania. Bipolar Disorders, 7, 68 –76.[CrossRef][Medline]
- First, M. B., Spitzer, R. L., Gibbon, M., et al (1995) Structured Clinical Interview for DSM–IV Axis Disorders – Patient Edition. New York: New York State Psychiatric Institute.
- Gitlin, M. J., Swendsen, J., Heller, T. L., et al
(1995) Relapse and impairment in bipolar disorder.
American Journal of Psychiatry,
152, 1635
–1640.
[Abstract/Free Full Text] - Goodnick, P. J., Fieve, R. R., Schlegel, A., et al
(1987) Predictors of interepisode symptoms and relapse in
affective disorder patients treated with lithium carbonate.
American Journal of Psychiatry,
144, 367
–369.
[Abstract/Free Full Text] - Gopal, S., Steffens, D. C., Kramer, M. L., et al (2005) Symptomatic remission in patients with bipolar mania: results from a double-blind, placebo-controlled trial of risperidone monotherapy. Journal of Clinical Psychiatry, 66, 1016 –1020.
- Hamilton, M. (1959) The assessment of anxiety states by rating. British Journal of Medical Psychology, 32, 50 –55.[Medline]
- Joffe, R. T., MacQueen, G. M., Marriott, M., et al (2004) A prospective, longitudinal study of percentage of time spent ill in patients with bipolar I or bipolar II disorders. Bipolar Disorders, 6, 62 –66.[CrossRef][Medline]
- Judd, L. L. & Akiskal, H. S. (2003) The prevalence and disability of bipolar spectrum disorders in the US population: re-analysis of the ECA database taking into account subthreshold cases. Journal of Affective Disorders, 73, 123 –131.[CrossRef][Medline]
- Judd, L. L., Akiskal, H. S., Schettler, P. J., et al
(2002) The long-term natural history of the weekly
symptomatic status of bipolar I disorder. Archives of General
Psychiatry, 59, 530
–537.
[Abstract/Free Full Text] - Keck, P. E., McElroy, S. L., Havens, J. R., et al (2003) Psychosis in bipolar disorder: phenomenology and impact on morbidity and course of illness. Comprehensive Psychiatry, 44, 263 –269.[CrossRef][Medline]
- Keitner, G. I., Solomon, D. A., Ryan, C. E., et al (1996) Prodromal and residual symptoms in bipolar I disorder. Comprehensive Psychiatry, 37, 362 –367.[CrossRef][Medline]
- Keller, M. B., Lavori, P. W., Kane, J. M., et al
(1992) Subsyndromal symptoms in bipolar disorder. A
comparison of standard and low serum levels of lithium. Archives of
General Psychiatry, 49, 371
–376.
[Abstract/Free Full Text] - Kessing, L. V., Hansen, M. G., Andersen, P. K., et al (2004) The predictive effect of episodes on the risk of recurrence in depressive and bipolar disorders – a life-long perspective. Acta Psychiatrica Scandinavica, 109, 339 –344.[CrossRef][Medline]
- MacQueen, G. M., Marriott, M., Begin, H., et al (2003) Subsyndromal symptoms assessed in longitudinal, prospective follow-up of a cohort of patients with bipolar disorder. Bipolar Disorders, 5, 349 –355.[CrossRef][Medline]
- Post, R. M., Denicoff, K. D., Leverich, G. S., et al (2003) Morbidity in 258 bipolar outpatients followed for 1 year with daily prospective ratings on the NIMH life chart method. Journal of Clinical Psychiatry, 64, 680 –690.
- Swann, A. C., Daniel, D. G., Kochan, L. D., et al (2004) Psychosis in mania: specificity of its role in severity and treatment response. Journal of Clinical Psychiatry, 65, 825 –829.
- Tohen, M., Waternaux, C. M. & Tsuang, M. T.
(1990a) Outcome in mania. A 4-year prospective
follow-up of 75 patients utilizing survival analysis. Archives of
General Psychiatry, 47, 1106
–1111.
[Abstract/Free Full Text] - Tohen, M., Waternaux, C. M., Tsuang, M. T., et al (1990b) Four-year follow-up of twenty-four first-episode manic patients. Journal of Affective Disorders, 19, 79 –86.[CrossRef][Medline]
- Tohen, M., Hennen, J., Zarate, C. M., et al
(2000) Two-year syndromal and functional recovery in 219
cases of first-episode major affective disorder with psychotic features.
American Journal of Psychiatry,
157, 220
–228.
[Abstract/Free Full Text] - Tohen, M., Zarate, C. A., Hennen, J., et al
(2003) The McLean–Harvard First-Episode Mania Study:
prediction of recovery and first recurrence. American Journal of
Psychiatry, 160, 2099
–2107.
[Abstract/Free Full Text] - Tohen, M., Greil, W., Calabrese, J. R., et al
(2005) Olanzapine versus lithium in the maintenance treatment
of bipolar disorder: a 12-month, randomized, double-blind, controlled clinical
trial. American Journal of Psychiatry,
162, 1281
–1290.
[Abstract/Free Full Text] - Yatham, L. N., Binder, C., Kusumakar,V., et al (2004) Risperidone plus lithium versus risperidone plus valproate in acute and continuation treatment of mania. International Clinical Psychopharmacology, 19, 103 –109.[CrossRef][Medline]
- Young, R. C., Biggs, J. T., Ziegler, V. E., et al
(1978) A rating scale for mania: reliability, validity and
sensitivity. British Journal of Psychiatry,
133, 429
–435.
[Abstract/Free Full Text]
This article has been cited by other articles:
 |
|
 |
|
  L. L. Judd, P. J. Schettler, H. S. Akiskal, W. Coryell, A. C. Leon, J. D. Maser, and D. A. Solomon Residual Symptom Recovery From Major Affective Episodes in Bipolar Disorders and Rapid Episode Relapse/Recurrence Arch Gen Psychiatry, April 1, 2008; 65(4): 386 - 394. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||