International Journal of Stress Management

© 2023 American Psychological Association 2023, Vol. 30, No. 4, 366–375

ISSN: 1072-5245 https://doi.org/10.1037/str0000296

Detecting Burnout: Identifying Key Symptoms Using

Standard and Machine Learning Methods
Gabriela Tavella, Michael Spoelma, and Gordon Parker
Discipline of Psychiatry and Mental Health, School of Clinical Medicine, The University of New South Wales

Individuals readily self-diagnose as experiencing burnout despite continuing debate among researchers and
practitioners regarding how the syndrome should best be defined and measured. The objective of this study
was to determine whether a new 34-item measure of burnout distinguished those who did and did not self-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

identify as burnt out and in doing so ascertain the most distinguishing symptoms. Six hundred twenty-five
This document is copyrighted by the American Psychological Association or one of its allied publishers.

participants recruited via Facebook completed the burnout measure online before reporting whether they
were currently experiencing burnout. Receiver operating characteristic analyses indicated that the measure
adequately discriminated between those who did (47.7%) and did not self-report burnout. Cutoff scores
based on Youden’s indices had comparable classificatory accuracy as prediction rule ensembles derived
through machine learning methods. Items capturing exhaustion, compromised cognition, lack of pleasure in
work, and self-criticism were the most distinguishing items across the analyses, while items depicting
empathy loss varied in their discriminatory capacity between analyses. Weighting symptom items according
to their discriminatory capacity did not improve classificatory accuracy compared to when all items were
weighted equally. Overall, the 34-item measure satisfactorily differentiated those with and without self-
reported burnout, with symptoms of exhaustion, cognitive dysfunction, lack of pleasure in work, and self-
criticism being most indicative of the syndrome. Future research is needed to validate the measure and its
cutoff values by comparing measure scores against those clinically judged as having burnout.

Keywords: burnout, depression, cognition, diagnosis, work

Supplemental materials: https://doi.org/10.1037/str0000296.supp

Public interest in burnout is at an all-time high, with studies Psychiatric Association, 2013), while it is listed in the International
reporting several serious consequences of the syndrome (Dyrbye Classification of Diseases, 11th Revision “not as a medical condi-
et al., 2019; Heinemann & Heinemann, 2017) and media outlets tion” (World Health Organization, 2019a), but rather as an
reporting stark increases in prevalence rates due to the COVID-19 “occupational phenomenon” and a “factor influencing health status”
pandemic (Aubusson, 2021; Hartley, 2020). Despite burnout’s “hot (World Health Organization, 2019b). The ICD’s triadic definition of
topic” status in public discourse, researchers and clinicians are yet to burnout is derived from the most widely used burnout measure, the
agree on the syndrome’s precise definition, its symptoms, how it is Maslach Burnout Inventory (MBI; Maslach et al., 2016), which
best measured, and whether it is in fact a distinct mental health defines burnout as a compilation of exhaustion, cynicism or
disorder that should be included in psychiatric diagnostic manuals depersonalization (i.e., loss of empathy), and decreased professional
(Bianchi et al., 2019; De Beer & Bianchi, 2017). We have conducted accomplishment. The MBI was not designed to be used as a
a series of exploratory studies with the objective of exploring some categorical diagnostic tool for determining cases and noncases of
of these issues and now report a study evaluating the discriminatory burnout. Rather, the MBI positions burnout as a dimensional
capacity of indicative symptoms. construct, with the authors stating that there is “no definitive score that
Burnout is not listed as a diagnosable disorder in the Diagnostic and ‘proves’ a person is ‘burned out’” (Maslach et al., 2016).
Statistical Manual of Mental Disorders, fifth edition (American We and others (Kristensen et al., 2005; Schaufeli et al., 2020; Tavella
et al., 2021; Tavella & Parker, 2020a) have previously questioned
whether the MBI’s three-factor burnout definition adequately
This article was published Online First May 25, 2023. operationalizes the syndrome. In an extensive review examining
Gordon Parker https://orcid.org/0000-0003-3424-5519 how burnout research has been conducted up until 2011, Heinemann
The study was supported by an Australian National Health and Medical and Heinemann (2017) concluded that “to date, most burnout research
Research Council (NHMRC; Grant GNT1176689). The contents of the has been circular, because it relies on questionnaires that measure
published material are solely the responsibility of the individual authors and symptoms that have not (yet) been clearly defined or unanimously
do not reflect the views of the NHMRC. The authors declare that they have no agreed upon in medicine and psychology”. They argued that future
competing interests.
burnout research should focus on determining an accurate burnout
The data used in this study have not been previously published or
disseminated elsewhere.
definition not based on previously established burnout measures and
Correspondence concerning this article should be addressed to Gordon that research should focus on elucidating whether burnout can be
Parker, Discipline of Psychiatry and Mental Health, School of Clinical distinguished from other psychological conditions. This latter point is of
Medicine, The University of New South Wales, Level 1 AGSM Building, particular relevance when considering the ongoing debate as to whether
UNSW, Sydney, NSW 2052, Australia. Email: [email protected] burnout is or is not synonymous with depression (Bianchi et al., 2015).

366
 DETECTING BURNOUT 367

Considering such issues, we recently undertook two studies to assess what members of the lay community perceive to be the most
reexamine the key features of burnout as experienced by the lay prominent features of burnout. Self-identification of burnout has
population and to examine its distinction from depression. The first been previously employed to examine burnout symptoms and
study involved 1,016 participants (self-identifying as experiencing correlates (Brady et al., 2022; Kavalieratos et al., 2017; Olson
burnout) completing a questionnaire that included a 106-item list et al., 2019; Pick & Leiter, 1991; Rohland et al., 2004; Sinsky
of putative burnout symptoms and some personality descriptors et al., 2021), with such studies finding that self-diagnosed burnout
(Tavella et al., 2020). The items were derived from previous corresponds with scores indicative of burnout on the MBI
burnout measures, a detailed review of the burnout literature and emotional exhaustion subscale. Iterative analyses of data from two
clinical experience. A bifactor solution with a general factor samples of participants with self-diagnosed burnout (Tavella et al.,
(representing variance in the data accounted for by the primary 2020, 2021) therefore allowed us to derive our current burnout model
dimension being measured) and three additional “specific” factors that conceptualizes burnout as a syndrome comprised of exhaustion,
(accounting for variance not captured by the general factor) was cognitive dysfunction, empathy loss, reduced work performance, and
judged as most adequately capturing the data, with the general social withdrawal, as summarized in Figure 1 and represented by our
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

factor weighting exhaustion, cognitive dysfunction, worry, depres- 34-item measure.

This document is copyrighted by the American Psychological Association or one of its allied publishers.

sion, and social withdrawal. The three specific factors captured (a) Validation of our measure ideally requires comparison of total
social withdrawal and disconnection, (b) cognitive dysfunction scores against those with definitive burnout diagnoses. Such an
coupled with reduced work efficacy, and (c) perfectionistic and objective is complex, however, as there currently exists no agreed
workaholism traits, with this last factor interpreted as suggesting upon definition and no formal diagnostic criteria for burnout, as just
personality styles that predispose to a burnout syndrome (and described. Therefore, before tackling such a task, we instead sought
allowing a diathesis–stress model rather than a symptom-based here to examine the extent to which our 34-item measure of burnout
model alone). distinguished those who did and did not self-identify as burnt out
A second study was then undertaken to assess (a) whether the and which symptoms were the most distinguishing. Strategically,
symptom clusters identified in the first study were evident in a second we wanted to assess whether the measure could detect burnout in
sample of participants self-identifying as being burnt out, (b) if participants blinded to the research objectives (i.e., they did not
burnout is best defined by a single symptom dimension/factor rather know when completing the questionnaire that burnout was the
than by multiple symptom dimensions, and (c) the status of depressive research topic of interest). We also sought to derive an optimal
symptoms in those experiencing burnout (Tavella et al., 2021). cutoff score for the 34-item measure differentiating those with and
Participants with self-identified burnout completed a questionnaire those without self-identified burnout.
that included a list of 137 candidate burnout symptoms, with Scoring the measure via a simple sum of scores on each of the 34
symptoms including those from the first study as well several items items has real-world advantages, being simple to score and interpret
from validated measures of depression. A series of factor analyses in practice. However, such scoring relies on the assumption that
resulted in a final bifactor solution made up of a 34-item general factor candidate symptoms effect a similar diagnostic loading, while it is
and five uncorrelated specific factors. The general factor was likely that some items are more discriminating than others. Thus,
dominated by items capturing exhaustion but cognitive dysfunction additional objectives were to determine the most distinguishing items
items were also prominent. Our general factor also included items between those who did and did not identify as burnt out, as well as
indicative of social withdrawal, irritability, excessive worrying, ascertain if weighting items from our measure according to their
empathy loss, and some depressive symptoms including low mood discriminatory capacity resulted in greater classificatory accuracy
and anhedonia. The specific factors were interpreted as encapsulating than when all items were weighted equally. These objectives were
(a) cognitive dysfunction, (b) empathy loss, (c) exhaustion, (d) approached in part by using machine learning (ML) methods, which
reduced work performance, and (e) social withdrawal in conjunction work to optimize the predictive performance of models and identify
with low mood. items or rules that are most “important” in optimizing that prediction
Our measure was derived from data provided by participants who (Fokkema & Strobl, 2020). ML techniques are being increasingly
self-identified as experiencing burnout. Using a “top-down” utilized in the fields of psychiatry and psychology because
approach, whereby existing definitions and/or measures of burnout they can optimize diagnosis, prognosis, and treatment recommenda-
would be used to determine eligible participants for inclusion, was tions (Dwyer et al., 2018) and were therefore employed in the
judged as inappropriate due to such a method creating a circularity present study.
issue. Indeed, Heinemann and Heinemann (2017) emphasized this
concern, highlighting that most burnout research uses “debatable
and contested constructs and inventories to identify participants who Method
purportedly suffer from burnout, and then examines this mental
Participants
condition in these individuals”. Second, there are no accepted
diagnostic criteria for burnout among practitioners/researchers, thus Individuals could participate if they were 18–65 years old, willing
burnout diagnosis cannot be standardized and it is currently up to to complete an anonymous online questionnaire, and fluent in
“the discretion of the doctor to assess a burnout diagnosis” (Korczak English. Participants were recruited via a Facebook advertisement
et al., 2010). Recognizing the widespread take up of the burnout which stated that the purpose of the study was to examine the
label by the lay community, and the associated distress reported by influence of personality and other emotional experiences on mental
individuals who self-diagnose (e.g., Turnbull & Rhodes, 2021), we health, and thus the participants were unlikely to be aware that the
have therefore employed a “bottom-up” approach across our studies study was examining burnout. The advertisement was presented to
in which those who self-identified as having burnout were studied to Facebook users located in Australia and stated that participants
 368 TAVELLA, SPOELMA, AND PARKER

Figure 1
Summary of Our New Symptom Model of Burnout
SPECIFIC FACTORS

Cognitive Dysfunction

Concentration issues,
memory and attention
problems

Exhaustion

Constant fatigue, no energy,

This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

not refreshed by sleep

This document is copyrighted by the American Psychological Association or one of its allied publishers.

GENERAL FACTOR
Social Withdrawal
Burnout
Keeping to oneself, not
Exhaustion, compromised cognition,
looking forward to seeing
withdrawal, irritability, worrying,
friends and family
low mood

Reduced Work Performance

Less productive, lower

quality of output, loss of
drive and pleasure from
work

Empathy Loss

Struggling to understand
others’ feelings, less
empathy towards people

could receive $10 (Australian dollar) or a shopping voucher of whatever reason, or their home/care duties if they were not formally
equivalent value in return for participating in the study. employed. Four questions presented after the 34-item measure asked
participants if they had ever been diagnosed with and/or treated for
depression and if they were currently experiencing or had ever
Procedure experienced burnout. At the end of the questionnaire, participants
The study was approved by the University of New South Wales provided details for remuneration.
Human Research Ethics Committee (UNSW HREC No. HC190213).
After providing online consent, participants completed demographic
Statistical Considerations
questions inquiring into age, gender, ethnicity, education level,
marital, and occupational status and then answered questions relating Reliability statistics (coefficient ω and hierarchical coefficient ωH)
to personality style. Participants were then presented with our 34-item calculated for our 34-item solution in our previous study (Tavella et
measure of burnout symptoms and asked to “please tick the degree to al., 2021) suggested that the general factor accounted for the
which the following features are presented for you currently and majority of variance in the sample. Measures derived from such
represent a change from your typical or ‘normal’ functioning,” with bifactor solutions where ωH is high for the general factor and low for
response categories being “not present,” “slightly,” “moderately,” and the specific factors may potentially be best represented as single-
“distinctly” (coded as 0, 1, 2, and 3, respectively). Some of the 34 scale scores based on the general factor alone, as subscales made up
items included reference to “work”; however, in acknowledging the of specific factor items “provide little unique, reliable variance”
possibility that some participants would be unemployed, we (Rodriguez et al., 2016). Thus, burnout as represented by our model
instructed participants to consider “work” to be their formal appears best measured using a single summed total score on a
employment/study if they were employed or a student, their most measure made up of our 34 items, and thus, a single total score was
recent employment if they had recently ceased employment for used in the current set of analyses.
 DETECTING BURNOUT 369

Analyses conducted in SPSS included t tests, chi-square (χ2) tests, educational attainment most often nominated was an undergradu-
and receiver operating characteristic (ROC) analyses, to determine ate degree (40.5%). Occupational statuses reported most
whether and to what degree a total summed score from the 34-item frequently were full-time employment (30.2%), part-time
measure discriminated between those who did and did not report employment (21.8%), and being a student (18.4%). The most
experiencing burnout. The area under the curve (AUC) generated for common occupations nominated were education professional
an ROC analysis is an indicator of discriminatory capacity, where (e.g., teacher, lecturer, tutor; 11%); clerical or administration
an AUC of 0.5 generally suggests no discrimination, 0.7–0.8 is worker (8.8%); retail, hospitality, or service manager (6.65%);
considered acceptable, 0.8–0.9 is considered excellent, and 0.9 or public servant (5.8%); and midwife or nurse (5.3%), while 8.3% of
greater is considered outstanding (Hosmer & Lemeshow, 2000). the sample recorded that they had never been employed in a
Cutoff values reported were those points where Youden’s index was traditional occupation or workplace.
maximized (sensitivity + specificity − 1; Youden, 1950).
The ML analyses were completed in R, using the pre package
Discrimination Between Burnout and
(Fokkema, 2020). The ML analyses used prediction rule ensembles
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

No Burnout Groups
(PREs; Fokkema & Strobl, 2020), which derive a set of “rules” from
This document is copyrighted by the American Psychological Association or one of its allied publishers.

the independent variables that consist of either individual (e.g., for a Of the 625 participants, 298 (47.7%) answered “yes” to the
variable v, v1 = yes) or multiple variables (e.g., v1 = yes and v2 = question “Are you currently experiencing burnout?” (hereinafter
yes and v3 = no). PRE output takes the form of a regression model labeled as the “burnout group”), with the 327 participants answering
such that each rule has a regression weight, which only contributes “no” labeled as the “no burnout group.” There were no significant
to the overall regression output if the rule is true. PREs have been differences in age, gender, ethnicity, or educational attainment level
established as an ML technique that strikes an acceptable balance between the two groups. There was, however, a significant difference
between more easily interpretable techniques, such as single in occupational status (χ2 = 17.8, p = .007). Examination of adjusted
decision trees, and more accurate but complex techniques, such as residuals suggested differences between the two groups in the
random forests (Fokkema & Strobl, 2020). percentage of those unemployed (12.8% of the burnout group and
Data from the entire sample served as the “training” data through 6.1% of the no burnout group) and those retired (5.4% of the burnout
which the PRE was derived. The independent variables were group and 12.2% of the no burnout group). Tests for log-odds ratios
subjects’ responses to the 34 items, with the dependent variable indicated that both differences were significant (difference in
being burnout “caseness” as self-reported by participants (i.e., percentage unemployed, p = .008; difference in percentage retired,
burnout or no burnout). The outputs of the regression models were p = .001).
used to calculate the probability that an individual belonged to
the “burnout” class (as opposed to the “no burnout” class). Thus, a
Mean Differences
positive regression output coefficient corresponded to burnout
diagnosis, and a negative output coefficient corresponded to a The highest possible score on the nondichotomized 34-item
no burnout diagnosis. Predictive accuracies of the PRE were measure when each item is equally weighted is 102. The burnout and
calculated by comparing self-reported burnout caseness to the no burnout groups returned mean scores of 53.7 and 27.8,
PRE diagnostic predictions from “testing” data derived from respectively, which differed significantly (t = 15.4, p > .001,
performing 10 repetitions of 10-fold cross-validation (Fokkema & SE = 1.68). All 34 items are listed in Table 1. There were significant
Strobl, 2020). differences in the reported frequency of all items between the
The output of the ML analyses also generated relative “importance” burnout and no burnout groups (smallest χ2 = 32.1, p > .001), with
values (the absolute value of the regression coefficient, multiplied all items rated as more severe in the burnout group. To examine
by the standard deviation) of each rule and individual variable in which items had the greatest discriminatory capacity, mean
contributing to the associated PRE model, with larger importance differences in scores on each item (coded as 0, 1, 2, 3) were
values indicative of that rule or variable having greater effect on calculated, and the items are listed in Table 1 in descending order
the model. of mean difference value. The table indicates that the greatest
Two sets of analyses were completed: the first using the raw data differences in reporting of items between the burnout and no burnout
with responses coded 0–3 (from “not present” to “distinctly”) and groups were mostly those items indicative of exhaustion (e.g., “I feel
the second using the Likert data once it had been dichotomized, such emotionally drained and exhausted,” “I constantly feel tired or
that response categories originally coded as 0 (“not present”) or 1 fatigue”) and reduced cognitive function (e.g., “I’m finding it hard
(“slightly”) were recoded as 0 (i.e., “not present”) and response to concentrate on the task at hand,” “I’m feeling slowed down
categories originally coded as 2 (“moderately”) or 3 (“distinctly”) mentally”), as well as the items “I’ve started feeling more self-
were recoded as 1 (i.e., “present”). critical and hard on myself,” “I cannot get pleasure out of my work,”
and “I’m spending much of my days worrying.” Depressive items
indicating low mood and anhedonia were ranked in the middle of the
Results item set, while items denoting reduced empathy loss were the least
differentiating.
Demographics
Six hundred and twenty-five individuals completed the
ROC Analyses
questionnaire. The sample had a female preponderance (79.4%)
and a mean age of 44.35 years, with most identifying their ethnicity An ROC analysis of participants’ scores returned an AUC of .81,
as non-Indigenous Australian (57.8%). The highest level of which is indicative of excellent discrimination. A cutoff score of 37
 370 TAVELLA, SPOELMA, AND PARKER

Table 1 ML Analyses
Items From 34-Item Measure Ranked in Order From Most to Least
The analyses compared the burnout and no burnout groups through
Discriminatory Based on Mean Differences in Reported Response
a binary regression model. For both the nondichotomized and
Categories
dichotomized data, PRE-allocated diagnoses were compared to self-
Variable no. Item reported diagnoses. As shown in Table 2, PRE-derived diagnoses
offered overall slightly higher accuracy than the ROC analyses, with
v10 I feel emotionally drained and exhausted the overall classification rate of the dichotomized PRE slightly
v26 I constantly feel tired or fatigued
v3 I feel worn out superior to the nondichotomized PRE.
v11 I’ve experienced a loss of energy (making it hard to get The rules with associated coefficient and importance values for
going in the morning) the PRE analysis for both the nondichotomized and dichotomized
v12 I’ve started feeling more self-critical and hard on myself data are provided in the Supplemental Material, with a total of 23
v8 I’ve been waking up feeling tired
v14 I cannot get pleasure out of my work
rules generated for the nondichotomized data (which comprised 16
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

v9 I’m finding it hard to concentrate on the task at hand of the possible 34 items) and 17 rules generated for the dichotomized
This document is copyrighted by the American Psychological Association or one of its allied publishers.

v27 I’m feeling slowed down mentally (e.g., hard to find data (which comprised 15 of the possible 34 items). The individual
words, slowed thoughts) variable importance values for the items included in the generated
v13 I’m spending much of my days worrying rules are shown in Figure 2. For the nondichotomized data, some of
v7 I lack energy across the day
v20 I feel as though I am stagnating and life is passing me by the highest ranking items were those capturing exhaustion, cognitive
v23 I have to reread things because I was not concentrating performance issues, and not getting pleasure out of one’s work.
the first time Items depicting self-criticism (“I’ve started feeling more self-critical
v17 I’m finding it more difficult to take life as it comes and hard on myself”), empathy loss (e.g., “I am less empathetic than
v2 I feel sad, empty and hopeless
v31 My attention is less focussed than usual
usual,” “I care less about what happens to people that I work with”),
v19 I no longer look forward to things that would normally and reduced performance and motivation (i.e., “The quality of my
give me pleasure work output is lower than usual” “I’ve stopped feeling very driven”)
v16 I cannot concentrate or register new information because had some importance, while low mood and anhedonia items were
of foggy thinking absent from the 16 items.
v29 I’m taking longer to finish tasks at work
v28 The quality of my work output is lower than usual Items’ importances generated for the dichotomized data were similar
v22 I’m finding little things and chores frustrating to those of the nondichotomized data for many of the exhaustion items,
v21 I feel like I am making less of a difference at work than as well as the item “I cannot get pleasure out of my work.” The self-
usual criticism item was ranked as the seventh most important item, one item
v18 I’ve started keeping to myself
v25 I am less productive at work than usual
denoting empathy loss (“I am struggling to understand the feelings of
v33 I’ve stopped feeling very driven to keep meeting my colleagues, customers and/or recipients of my care”) was ranked 11th,
responsibilities while the item depicting low mood was ranked 12th.
v1 I am not refreshed by sleep
v24 I’ve stopped looking forward to spending time with
friends and family
v6 My work performance has worsened
Discussion
v15 I’ve withdrawn from family and friends Acknowledging the debate as to how burnout is best defined, we
v5 My capacity to remember things is not as good as usual
v34 I care less about what happens to people that I work with have recently developed our own conceptualization of the syndrome
(e.g., colleagues, customers, recipient of my care) than as one comprised of exhaustion, cognitive dysfunction, empathy loss,
usual reduced work performance and social withdrawal, and quantified by a
v30 I am less empathetic than usual 34-item measure. The present study first aimed to evaluate the extent
v4 I am struggling to understand the feelings of colleagues,
customers and/or recipients of my care
to which our measure could distinguish between those with and
v32 I feel less empathy and sympathy toward people in without self-defined burnout and derive a cutoff score. Total scores
general than usual on our 34-item measure (with all items weighted equally) were able to
distinguish between those who were and were not experiencing self-
reported burnout, with AUC for both analyses indicating that the
measure had satisfactory discriminatory capacity. Cutoff scores of 37
or more as being indicative of self-reported burnout was determined or more for the nondichotomized measure and 11 or more for the
by examining Youden’s indices. This cutoff value had a sensitivity dichotomized data as indicative of burnout had moderate overall
of 75.5% and a specificity of 70.9%, with an overall classification classification accuracies (73.1% and 72.5%, respectively).
accuracy of 73.1% (see Table 2). As noted, the authors of the MBI have not imposed cutoff scores
The ROC analysis was repeated with the data dichotomized (i.e., for the MBI. Such a position is understandable as the MBI was not
symptom present vs. not present). The dichotomized ROC curve had designed to be used as a diagnostic tool for determining burnout
an AUC of .80, and a cutoff of 11 or more out of a possible score caseness, but rather as a research tool. Nonetheless, categorical
of 34 being indicative of burnout was identified by Youden’s measures with thresholds for determining an individual’s caseness
indices. This cutoff score had a sensitivity of 74.2% and a specificity are useful for calculating accurate prevalence rates and enhancing
of 70.9%, with an overall classification accuracy of 72.5% (as the clinical management of conditions that require disorder-specific
shown in Table 2) and thus had comparable classification accuracy interventions (Bianchi et al., 2018). Some have derived cutoff scores
to the cutoff determined for the nondichotomized data. for the MBI for such diagnostic purposes (Schaufeli et al., 2009), but
 DETECTING BURNOUT 371

Table 2
Sensitivity, Specificity, and Overall Classification Accuracy of Receiver Operating Characteristic (ROC) and Machine
Learning (ML) Models

ROC analyses ML analyses

Classification accuracy Nondichotomized data Dichotomized data Nondichotomized data Dichotomized data

Sensitivity 75.5% 74.2% 76.51% 81.21%

Specificity 70.9% 70.9% 71.25% 69.72%
Overall accuracy 73.1% 72.5% 73.76% 75.20%

these scores have been criticized by other researchers for having been measure in picking up true burnout cases, as well as—and perhaps
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

chosen arbitrarily and not necessarily congruent across studies more importantly when considering the debated overlap between
This document is copyrighted by the American Psychological Association or one of its allied publishers.

(Bianchi et al., 2018). Cutoff scores used for any burnout measure burnout and depression—the specificity of the measure in discrimi-
should be standard across studies so to allow for across-study nating between burnout and other mental health conditions.
comparisons, with any such cutoff score requiring validation by Additional objectives of the present study were to evaluate which
examining its capacity to distinguish clinically observed cases of items best distinguished between the groups, as well as to determine
burnout. We acknowledge that such validation research is required for if weighting items according to their discriminatory capacity
the cutoff scores we report here for our measure. Such research should resulted in greater classificatory accuracy than when all items were
seek to determine which cutoff score maximizes the sensitivity of the weighted equally. Examination of the importance of individual

Figure 2
Individual Variable Importances for the Nondichotomized (Top Graph) and Dichotomized (Bottom Graph)
Data Prediction Rule Ensembles
 372 TAVELLA, SPOELMA, AND PARKER

variables in the ML analyses revealed key items pertinent to acknowledge that those with burnout are likely to experience
distinguishing participants with and without self-reported burnout. symptoms of depression, our results suggest that general anhedonia
Across both ML analyses, exhaustion items were consistently and low mood are not highly differentiating between those with and
ranked as among the most important (e.g., “I feel emotionally without self-identified burnout, as they are for depression, thus
drained and exhausted,” “I feel worn out,” “I constantly feel tired or rejecting the proposition that depression and burnout are basically
fatigued”), with such items also the most distinguishing items when synonymous.
ranked in terms of mean differences in response categories reported. Another symptom of depression is a feeling of worthlessness,
This prominence is not surprising, as it is widely accepted that which may be represented to some degree by our item “I’ve started
“exhaustion is the central quality of burnout and the most obvious feeling more self-critical and hard on myself.” This item was rated as
manifestation of this complex syndrome” (Maslach et al., 2001). moderately to highly distinguishing in both the mean differences
While the pertinence of exhaustion in burnout is not debated, and ML analyses. While we have previously reported reduced self-
questions remain as to what additional symptoms are imperative to worth to be more prominent in depression than burnout (Tavella &
defining the syndrome. The other two items ranked as highly Parker, 2020b), in that study, we also noted that most participants
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

discriminating across both ML analyses were the items “I have to who did report burnout as effecting their self-esteem specified that
This document is copyrighted by the American Psychological Association or one of its allied publishers.

reread things because I was not concentrating first time” and “I cannot this was due to their work performance being integral to their
get any pleasure out of my work.” The former we have identified as identity, so challenges in the work environment due to burnout
a compromised cognition item (Tavella et al., 2021), with a few resulted in subsequent reductions in their self-worth. Interestingly
compromised cognitive items ranked among the most distinguishing here, however, other items that were specific to work outcomes,
in the mean differences’ analysis (i.e., v9, v27, v23). Cognitive specifically those relating to reduced productivity or motivation
symptoms are not included in the MBI, but we have argued for their (e.g., v33, v21, v25, v6), did not rank highly as distinguishing
significance in our previous quantitative and qualitative burnout burnout in the mean differences analysis nor were many included in
studies (Tavella et al., 2020, 2021; Tavella & Parker, 2020a). Other 16 most important items in either ML analyses. Other researchers
studies have also identified cognitive problems in those with burnout have questioned the centrality of reduced professional accomplish-
(Beck et al., 2013; Sandström et al., 2005), providing weight to the ment to burnout, arguing that the relevant MBI subscale may be
argument that impaired cognition is one of burnout’s “cardinal measuring a consequence of burnout rather than a key symptom
symptoms” (Bianchi et al., 2015). construct (Bianchi et al., 2018; Kristensen et al., 2005; Schaufeli &
Of note, the lack of pleasure item ranked as important in the ML Van Dierendonck, 1993). We have hypothesized previously that
analyses as well as highly discriminatory in the mean differences reduced work output may be secondary to cognitive impairment
analysis was context-bound to work only (i.e., “I cannot get pleasure related to burnout, rather than inherent to burnout itself (Tavella
out of my work”). The context-free anticipatory anhedonia item “I et al., 2020). Future longitudinal studies assessing the developmen-
no longer look forward to things that would normally give me tal trajectory of cognitive and work performance issues in burnout
pleasure” was not ranked as important in either ML analysis and was are needed to test this hypothesis.
only moderately distinguishing as the 17th item when ranked in the In the ML analyses, items depicting empathy loss (“I am less
mean differences analysis. While anhedonia is a key feature of empathetic than usual,” “I care less about what happens to people that I
depression (being one of two required criteria for major depressive work with” for the nondichotomized data and “I am struggling to
disorder in the Diagnostic and Statistical Manual of Mental understand the feelings of colleagues, customers and/or recipients of
Disorders, fifth edition), the context specificity of the distinguishing my care” for the dichotomized data) had some differential importance,
anhedonia item here aligns somewhat with our previous qualitative while these items were among the lowest ranking items in the mean
study (Tavella & Parker, 2020b) and with such differentiation of differences analysis. It is difficult to determine why the empathy loss
distinct importance in differentiating burnout from depression. In items varied in their discriminatory ranking between our two types of
that study, participants reported that the most distinguishing feature analyses, but we have previously queried whether the centrality of
of their burnout when compared to their experiences of depression empathy loss to burnout may only be evident in those working in the
was that the former was “work-related” (irrespective of whether caregiving/service professions (Tavella et al., 2021). We cannot
“work” was formal/paid employment or home/care duties), while confirm this interpretation here as we did not examine differences in
the latter was more generalized across contexts. While such a result item affirmation based on occupation type in the current analysis.
is perhaps to be expected, we also highlight that considering burnout Nonetheless, Maslach and colleagues seemingly indirectly acknowl-
only as it pertains to work may be too simplistic. Indeed, other edged this possibility when they adapted the empathy loss subscale
evidence indicates burnout can occur outside of formal employment (i.e., depersonalization) of the MBI into a “cynicism” scale to capture
contexts, such as in parenting (Roskam et al., 2017), while one study feelings of detachment from the work environment, rather than loss of
found that almost half of those with burnout symptoms attribute empathy toward service recipients, for those outside of the human
their symptoms to factors not related to their jobs (Bianchi & services industry (Maslach et al., 2016).
Brisson, 2019). The contextual nuances of burnout (especially in The ML analyses generated a distinct set of rules for both the
comparison to those of depression) evidently remain to be nondichotomized and dichotomized PREs. The PREs had satisfac-
elucidated in future studies. tory classificatory accuracy but the individual rules computed by the
The other key DSM symptom of depression included in our analyses are complex and, in practice, would require importing
measure—the item capturing low mood and hopelessness (v2)—was patient data into a computer program to generate classification
not included in the 16 most important items in the nondichotomized information. In contrast, scores on the measure when all items were
ML analyses, while it was ranked 12th in the dichotomized ML equally weighted can be calculated easily, and the cutoff scores
analysis and 15th in the mean differences’ analysis. Thus, while we generated from the ROC analyses using such equally weighted item
 DETECTING BURNOUT 373

data had comparable classificatory accuracy to the PREs. This the research literature, for the general public, “there seems to be no
finding suggests that, in practice, there is no need for differential doubt that burnout is a real and serious phenomenon”, and thus, it is
symptom item weightings, making scoring of the measure simpler. not surprising that our participants were seemingly readily able to
While not a key objective of the study, we note differences in nominate whether they believed they were experiencing burnout. In
demographics of our groups. It is not surprising that a greater attempting to understand burnout as it is experienced by the lay
proportion of the “no burnout” group reported being retired, as those population, and acknowledging the lack of consensus across
who are retired would not be exposed to external work stressors researchers as to how burnout should be defined, we elected to
known to trigger burnout, such as work overload and lengthy work use self-diagnosis to understand how burnout is experienced by
hours (Sharma & Cooper, 2016). Unsurprisingly, burnout has been members of the lay community and to construct a “bottom-up”
previously associated with greater rates of intention to leave and/or definition of the syndrome based on such experiences. As mentioned
quitting one’s job (Lambert et al., 2010), and thus, while speculative, previously, burnout self-diagnosis has been used in past studies to
the higher unemployment rate in the burnout group here may have examine burnout symptoms and correlates (Kavalieratos et al., 2017;
been due to participants in this group being more likely to have left Olson et al., 2019; Sinsky et al., 2021). Nonetheless, and as
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

work due to burnout. We note that in an earlier study, we found that mentioned, the validity of our burnout model and the key symptoms
This document is copyrighted by the American Psychological Association or one of its allied publishers.

having ceased work due to burnout (compared to identifying as burnt identified here should be examined in future research that features
out but still working) did not influence the burnout symptoms systematic clinical observation to assess for a primary diagnosis of
reported by participants (Tavella et al., 2020). burnout as against any other condition (Bianchi et al., 2015).
Another limitation is the representativeness of our sample.
Participants were predominantly female, which presumably reflects
Theoretical and Practical Implications
women being more likely to participate in health research than
The present study advances our previous burnout research by men (Rourke & Lakner, 1989). Furthermore, most of the sample
identifying which of the burnout symptoms we have previously identified as non-Indigenous Australian, which suggests a Caucasian
identified appear to be most pertinent for sufferers. Most notably, our preponderance. The sample was also primarily tertiary-educated.
findings support the centrality of exhaustion to burnout, while also While most Australian adults do have a bachelor’s degree or higher
highlighting compromised cognition as another potential key symptom. (Australian Bureau of Statistics, 2017), such a sampling composition
Considering the key debate as to whether burnout is synonymous with may mask differences in burnout experienced by individuals based
depression, our results here highlight that those symptoms definitional on education level. Thus, the generalizability of our results should be
to experiences of depression are not necessarily definitional to interpreted with caution. In addition, while we did specify that items
individuals experiencing burnout, thus lending support to the argument relating to “work” in the questionnaire could include activities that
that burnout and depression are not synonymous states. occur outside of the formal employment context (i.e., home/care
Our burnout model, as represented by the 34-item measure, responsibilities), the different rates of unemployment between the
expands the definition of burnout beyond the three-factor two groups may have influenced responses to the work-related items
conceptualization that underpins the MBI. While we noted in the in the questionnaire. Excluding participant who were not employed
introduction that this study is part of a first step in validating our may have clarified this issue; however, we note both groups
model, we have begun further validation of the measure with one had approximately 12% of members not currently working (i.e.,
unpublished study supporting the convergent validity of the measure unemployed in the burnout group and retired in the no burnout
as compared to the MBI and another unpublished study evaluating group) and so had an equal proportion of participants responding to
its specificity in measuring cases of burnout as against cases of work-related items while not being currently employed.
clinical depression, with such validation studies required before In relation to the ML analyses, we tested and trained the model on
widespread use of the measure in clinical and/or research contexts. the same data, despite it being preferable to use different data sets for
While we were able to identify which burnout symptoms may be training and testing. To somewhat account for this limitation, we used
best differentiating, a key finding of practical importance is that cross-validation to get a more realistic and generalizable estimate of
weighting symptom items according to their discriminatory capacity the accuracy, with this method being appropriate when a separate
did not improve classificatory accuracy. Being able therefore to score a test set is not available (James et al., 2021) and thus considered
measure using a simple sum of scores on each item is advantageous as it a reasonable approach when using small data sets common in
requires less computational effort and can be done by hand. psychiatric research.
Furthermore, dichotomizing the items as either “present” or “absent” The study was strengthened by the absence of any mention of
was just as accurate as preserving the four response categories in burnout during study recruitment, or before or during the presentation
classifying participants across the ROC and ML analyses. This finding of the 34-item measure under examination. Participants were therefore
again has practical advantages by reducing the burden on those likely blinded to the intended purpose of the measure, thus reducing
completing the questionnaire, as it suggests that identifying a symptom the potential for any response bias effected by demand characteristics.
as present is likely sufficient rather than having to both evaluate its Further, we completed all analyses on two sets of data—one in which
presence and then grade its severity. the data included all response categories and another where the data
were dichotomized (i.e., symptom present or absent). Analyses of the
two data sets demonstrated comparable overall accuracy, suggesting
Limitations and Strengths
that use of the measure with dichotomized response categories
We now acknowledge study limitations. We examined cases and (i.e., symptom present or not) is to be recommended, being easier
noncases of burnout based on self-report. Heinemann and Heinemann to complete by individuals and interpret by researchers and clinicians
(2017) noted that despite burnout’s ambiguous status and definition in in practice.
 374 TAVELLA, SPOELMA, AND PARKER

Conclusion Fokkema, M., & Strobl, C. (2020). Fitting prediction rule ensembles to
psychological research data: An introduction and tutorial. Psychological
Overall, the present study identifies several psychological symptoms Methods, 25(5), 636–652. https://doi.org/10.1037/met0000256
associated with experiences of burnout, with exhaustion, compromised Hartley, A. (2020). Women enduring higher rates of anxiety and depression
cognition, lack of pleasure in work, and self-criticism being distinctive. during COVID-19 pandemic. ABC News. https://www.abc.net.au/news/
Further, while burnout may commonly be accompanied by anticipa- 2020-11-17/stressed-out-women-burnt-out-covid-19-coronavirus-pandemi
tory anhedonia and low mood (both key symptoms of depression), c/12847154
such symptoms are less prominent in distinguishing between those Heinemann, L. V., & Heinemann, T. (2017). Burnout research: Emergence
who do and do not report being burnt out. The study offers a first step in and scientific investigation of a contested diagnosis. SAGE Open, 7(1), 1–
establishing whether our new model of burnout, represented here in our 12. https://doi.org/10.1177/2158244017697154
Hosmer, D., & Lemeshow, S. (2000). Applied logistic regression (2nd ed.).
34-item measure, can distinguish between those who are and are not
Wiley. https://doi.org/10.1002/0471722146
experiencing burnout. We judge that further evaluation of the measure
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to
and its cutoff values in clinical burnout samples in future research statistical learning with applications in R (2nd ed.). Springer. https://
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

would be beneficial. doi.org/10.1007/978-1-0716-1418-1

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Kavalieratos, D., Siconolfi, D. E., Steinhauser, K. E., Bull, J., Arnold, R. M.,
Swetz, K. M., & Kamal, A. H. (2017). “It is like heart failure. It is
References chronic … and it will kill you”: A qualitative analysis of burnout among
hospice and palliative care clinicians. Journal of Pain and Symptom
American Psychiatric Association. (2013). Diagnostic and statistical manual
Management, 53(5), 901–910.e1. https://doi.org/10.1016/j.jpainsymman
of mental disorders (5th ed.). https://doi.org/10.1176/appi.books.97808
.2016.12.337
90425596
Korczak, D., Huber, B., & Kister, C. (2010). Differential diagnostic of the
Aubusson, K. (2021). Burnout warning for GPs ahead of mass vaccination
burnout syndrome. GMS Health Technology Assessment, 6, Article Doc09.
drive. The Sydney Morning Herald. https://www.smh.com.au/national/bu
https://doi.org/10.3205/hta000087
rnout-warning-for-gps-ahead-of-mass-vaccination-drive-20210301-p576u
Kristensen, T. S., Borritz, M., Villadsen, E., & Christensen, K. B. (2005).
c.html
Australian Bureau of Statistics. (2017). 2016 Census QuickStats. https://qui The Copenhagen Burnout Inventory: A new tool for the assessment of
ckstats.censusdata.abs.gov.au/census_services/getproduct/census/2016/qui burnout. Work and Stress, 19(3), 192–207. https://doi.org/10.1080/02678
ckstat/036 370500297720
Beck, J., Gerber, M., Brand, S., Pühse, U., & Holsboer-Trachsler, E. (2013). Lambert, E. G., Hogan, N. L., & Altheimer, I. (2010). An exploratory
Executive function performance is reduced during occupational burnout but examination of the consequences of burnout in terms of life satisfaction,
can recover to the level of healthy controls. Journal of Psychiatric Research, turnover intent, and absenteeism among private correctional staff. The
47(11), 1824–1830. https://doi.org/10.1016/j.jpsychires.2013.08.009 Prison Journal, 90(1), 94–114. https://doi.org/10.1177/0032885509
Bianchi, R., & Brisson, R. (2019). Burnout and depression: Causal 357586
attributions and construct overlap. Journal of Health Psychology, 24(11), Maslach, C., Jackson, S. E., & Leiter, M. P. (2016). Maslach Burnout
1574–1580. https://doi.org/10.1177/1359105317740415 Inventory (4th ed.). Mind Garden.
Bianchi, R., Schonfeld, I. S., & Laurent, E. (2015). Burnout-depression Maslach, C., Schaufeli, W. B., & Leiter, M. P. (2001). Job burnout. Annual
overlap: A review. Clinical Psychology Review, 36, 28–41. https://doi.org/ Review of Psychology, 52(1), 397–422. https://doi.org/10.1146/annurev
10.1016/j.cpr.2015.01.004 .psych.52.1.397
Bianchi, R., Schonfeld, I. S., & Laurent, E. (2018). Burnout syndrome and Olson, K., Sinsky, C., Rinne, S. T., Long, T., Vender, R., Mukherjee, S.,
depression. In Y.-K. Kim (Ed.), Understanding depression: Volume 2. Bennick, M., & Linzer, M. (2019). Cross-sectional survey of workplace
Clinical manifestations, diagnosis and treatment (pp. 187–202). Springer. stressors associated with physician burnout measured by the Mini-Z and
https://doi.org/10.1007/978-981-10-6577-4_14 the Maslach Burnout Inventory. Stress and Health, 35(2), 157–175.
Bianchi, R., Schonfeld, I. S., & Laurent, E. (2019). Burnout: Moving beyond https://doi.org/10.1002/smi.2849
the status quo. International Journal of Stress Management, 26(1), 36–45. Pick, D., & Leiter, M. P. (1991). Nurses’ perceptions of burnout: A comparison
https://doi.org/10.1037/str0000088 of self-reports and standardized measures. Canadian Journal of Nursing
Brady, K. J. S., Ni, P., Carlasare, L., Shanafelt, T. D., Sinsky, C. A., Linzer, Research, 23(3), 33–48. https://cjnr.archive.mcgill.ca/article/view/1121
M., Stillman, M., & Trockel, M. T. (2022). Establishing crosswalks between Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Applying bifactor
common measures of burnout in US physicians. Journal of General Internal statistical indices in the evaluation of psychological measures. Journal of
Medicine, 37(4), 777–784. https://doi.org/10.1007/s11606-021-06661-4 Personality Assessment, 98(3), 223–237. https://doi.org/10.1080/00223891
De Beer, L. T., & Bianchi, R. (2017). Confirmatory factor analysis of the .2015.1089249
Maslach Burnout Inventory: A Bayesian structural equation modeling Rohland, B. M., Kruse, G. R., & Rohrer, J. E. (2004). Validation of a single-item
approach. European Journal of Psychological Assessment, 35(2), 217– measure of burnout against the Maslach Burnout Inventory among
224. https://doi.org/10.1027/1015-5759/a000392 physicians. Stress and Health, 20(2), 75–79. https://doi.org/10.1002/smi.1002
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018). Machine learning Roskam, I., Raes, M.-E., & Mikolajczak, M. (2017). Exhausted parents:
approaches for clinical psychology and psychiatry. Annual Review of Development and preliminary validation of the Parental Burnout Inventory.
Clinical Psychology, 14(1), 91–118. https://doi.org/10.1146/annurev-cli Frontiers in Psychology, 8, Article 163. https://doi.org/10.3389/fpsyg.20
npsy-032816-045037 17.00163
Dyrbye, L. N., Shanafelt, T. D., Johnson, P. O., Johnson, L. A., Satele, D., & Rourke, D. O., & Lakner, E. (1989). Gender bias: Analysis of factors causing
West, C. P. (2019). A cross-sectional study exploring the relationship between male underrepresentation in surveys. International Journal of Public
burnout, absenteeism, and job performance among American nurses. BMC Opinion Research, 1(2), 164–176. https://doi.org/10.1093/ijpor/1.2.164
Nursing, 18(1), Article 57. https://doi.org/10.1186/s12912-019-0382-7 Sandström, A., Rhodin, I. N., Lundberg, M., Olsson, T., & Nyberg, L. (2005).
Fokkema, M. (2020). Fitting prediction rule ensembles with R package pre. Impaired cognitive performance in patients with chronic burnout syndrome.
Journal of Statistical Software, 92(12), 1–30. https://doi.org/10.18637/jss Biological Psychology, 69(3), 271–279. https://doi.org/10.1016/j.biopsycho
.v092.i12 .2004.08.003
 DETECTING BURNOUT 375

Schaufeli, W. B., Desart, S., & De Witte, H. (2020). Burnout Assessment Tavella, G., & Parker, G. (2020a). A qualitative reexamination of the key
Tool (BAT)-development, validity, and reliability. International Journal features of burnout. Journal of Nervous and Mental Disease, 208(6), 452–
of Environmental Research and Public Health, 17(24), Article 9495. 458. https://doi.org/10.1097/NMD.0000000000001155
https://doi.org/10.3390/ijerph17249495 Tavella, G., & Parker, G. (2020b). Distinguishing burnout from depression:
Schaufeli, W. B., Leiter, M. P., & Maslach, C. (2009). Burnout: 35 years of An exploratory qualitative study. Psychiatry Research, 291, Article 113212.
research and practice. Career Development International, 14(3), 204–220. https://doi.org/10.1016/j.psychres.2020.113212
https://doi.org/10.1108/13620430910966406 Turnbull, M. G., & Rhodes, P. (2021). Burnout and growth: Narratives of
Schaufeli, W. B., & Van Dierendonck, D. (1993). The construct validity of Australian psychologists. Qualitative Psychology, 8(1), 51–61. https://
two burnout measures. Journal of Organizational Behavior, 14(7), 631– doi.org/10.1037/qup0000146
647. https://doi.org/10.1002/job.4030140703 World Health Organization. (2019a). Burn-out an “occupational phenome-
Sharma, R. R., & Cooper, C. L. (2016). Executive burnout: Eastern and non”: International Classification of Diseases. https://www.who.int/ne
western concepts, models and approaches for mitigation. Emerald Group ws/item/28-05-2019-burn-out-an-occupational-phenomenon-international-
Publishing. https://doi.org/10.1108/9781786352859 classification-of-diseases
Sinsky, C. A., Brown, R. L., Stillman, M. J., & Linzer, M. (2021). World Health Organization. (2019b). International statistical classification
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

COVID-related stress and work intentions in a sample of US health care of diseases and related health problems (11th ed.).
This document is copyrighted by the American Psychological Association or one of its allied publishers.

workers. Mayo Clinic Proceedings: Innovations, Quality & Outcomes, Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35.
5(6), 1165–1173. https://doi.org/10.1016/j.mayocpiqo.2021.08.007 https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR28200301
Tavella, G., Hadzi-Pavlovic, D., & Parker, G. (2020). Burnout: Re- 06>3.0.CO;2-3
examining its key constructs. Psychiatry Research, 287, Article 112917.
https://doi.org/10.1016/j.psychres.2020.112917
Tavella, G., Hadzi-Pavlovic, D., & Parker, G. (2021). Burnout: Redefining Received December 4, 2022
its key symptoms. Psychiatry Research, 302, Article 114023. https:// Revision received March 16, 2023
doi.org/10.1016/j.psychres.2021.114023 Accepted April 21, 2023 ▪

E-Mail Notification of Your Latest Issue Online!

Would you like to know when the next issue of your favorite APA journal will be available
online? This service is now available to you. Sign up at https://my.apa.org/portal/alerts/ and you will
be notified by e-mail when issues of interest to you become available!