November 21, 2016

ARTICLE REVIEW

Sepsis, the syndrome of dysregulated inflammation that occurs with severe infection, is associated with high morbidity, mortality, and cost [1, 2]. In the USA, as well as in Europe, sepsis is the focus of national quality improvement programmes and a recent public reporting measure from the Centers for Medicare and Medicaid Services [3, 4].  To make the diagnosis of severe sepsis, clinicians must decide whether a patient has an infection, whether acute organ dysfunction is present, and whether acute organ dysfunction (when present) is attributable to infection. These determinations can be subjective and it is thus highly conceivable that thoughtful clinicians might differ substantially in their judgments. 
The authors intended to access and quantify clinicians / inter-observer variability in the diagnosis of sepsis as this has important implications for clinical care, epidemiologic and clinical studies, public health surveillance, pay-for-performance initiatives, and quality improvement programs.

For this study the investigators hypothesised that there would be significant variability in sepsis diagnoses, and that this variability would exist independent of physicians’ degree of confidence in their ability to apply the traditional consensus definitions of sepsis, and aimed to evaluate whether and to what degree intensivists agree in how they diagnose sepsis. The survey was conducted before the release of the new SCCM/ESICM consensus definition of sepsis [5].  

Five case vignettes of common scenarios of patients with suspected or confirmed infection and organ dysfunction were distributed to a sample of practicing intensivists. Respondents classified cases as systemic inflammatory response syndrome, sepsis, severe sepsis, septic shock, or none of the above. Inter-observer variability was calculated using Fleiss’ κ for the five-level classification, and for answers dichotomised as severe sepsis/septic shock versus not-severe sepsis/septic shock and any sepsis category (sepsis, severe sepsis, or septic shock) versus not-sepsis.

In the resulting article published in Critical Care (April 2016) [6], the authors reported five key findings:

  • Cases were felt to be extremely or very realistic in a great amount of respondents.
  • Almost all respondents felt strongly or somewhat confident in their ability to apply the traditional consensus sepsis definitions. 
  • Overall interrater agreement in sepsis diagnoses was poor.
  • When responses were dichotomised into severe sepsis/septic shock versus not-severe sepsis/septic shock or any sepsis category versus not-sepsis, agreement was still poor.
  • Among respondents who felt strongly confident in their ability to use sepsis definitions, agreement was no better.

Study Strengths & Limitations
The study has several strengths, namely: being the first study to examine variability in diagnosing sepsis by presenting identical cases to a group of intensivists, being used a fictional case vignettes felt to be very realistic and representative of common clinical scenarios (describing patients with suspected or documented infection and signs suggestive of organ dysfunction), and emphasising the importance of using objective measurements for diagnosing sepsis.

The limitations of this study mainly relate to: the relatively low response rate of the survey, the impossibility to compare the characteristics of physicians who were contacted but did not respond, the great variability in response rate to the survey in several states limiting the generalisability of findings. Furthermore, the fact of being conducted before the release of the new SCCM/ESICM consensus definition of sepsis, which may have performance characteristics in terms of inter-observer variability that are different from those of the prior sepsis definition set.

Take Home Messages

  • Diagnosing sepsis requires interpreting nonspecific signs, so there is a substantial amount of subjectivity in deciding whether infection is present, whether acute organ dysfunction is present, and whether acute organ dysfunction is attributable to infection. 
  • Inter-observer agreement among intensivists in diagnosing sepsis is poor. Even with the increased awareness and focus on sepsis in recent years, there is still a significant amount of variability in diagnosing sepsis among critical care physicians – the specialists who are generally felt to have the most expertise in caring for patients with sepsis. 
  • The present survey was conducted before the release of the new SCCM/ESICM consensus definition of sepsis, which uses the same framework of seeking patients with acute organ dysfunction attributable to infection, hence subjectivity in assigning sepsis diagnoses will likely persist. 
  • Subjectivity in diagnosing sepsis must be taken into account when interpreting the results of sepsis quality improvement initiatives and public reporting for sepsis bundle adherence, as well as for epidemiologic studies and clinical trials. 
  • Objective criteria and standardised methodology are needed to enhance consistency and comparability in sepsis research, surveillance, and quality reporting.

This article review was submitted by ESICM Journal Review Club member Carla Teixeira.

References

1. Minino AM, Murphy SL. Death in the United States, 2010. NCHS Data Brief. 2012; (99). 

2. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011. HCUP Statistical Brief #160. Rockville, MD: Agency for Healthcare Research and Quality; August2013. http://www.hcup-us.ahrq. gov/reports/statbriefs/sb160.pdf. Accessed 23 March 2016.

3. Centers for Medicare & Medicaid Services, Department of Health and Human Services. Medicare program; hospital inpatient prospective payment systems for acute care hospitals and the long-term care hospital prospective payment system and fiscal year 2015 rates; quality reporting requirements for specific providers; reasonable compensation equivalents for physician services in excluded hospitals and certain teaching hospitals; provider administrative appeals and judicial review; enforcement provisions for organ transplant centers; and electronic health record (EHR) incentive program. Final rule. Fed Regist. 2014;79(163):49853–50536. 

4. National Quality Forum. Severe sepsis and septic shock: management bundle (composite measure). http://www.qualityforum.org/Qps/QpsTool. aspx. Accessed 23 March 201.

5. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315(8):801–10.

6. Rhee C, Kadri SS, Danner RL, et al. Diagnosing sepsis is subjective and highly variable: a survey of intensivists using case vignettes. Critical Care. 2016;20:89. doi:10.1186/s13054-016-1266-9.

Comment on this news