OBJECTIVES: To determine the accuracy of predictions of dying at different cut-off thresholds and to acknowledge the extent of clinical uncertainty.
DESIGN: Secondary analysis of data from a prospective cohort study.
SETTING: An online prognostic test, accessible by eligible participants across the UK.
PARTICIPANTS: Eligible participants were members of the Association of Palliative Medicine. 99/166 completed the test (60%), resulting in 1980 estimates (99 participants × 20 summaries).
MAIN OUTCOME MEASURES: The probability of death occurring within 72 hours (0% certain survival-100% certain death) for 20 patient summaries. The estimates were analysed using five different thresholds: 50/50%, 40/60%, 30/70%, 20/80% and 10/90%, with percentage values between these extremes being regarded as 'indeterminate'. The positive predictive value (PPV), negative predictive value (NPV) and the number of indeterminate cases were calculated for each cut-off.
RESULTS: Using a <50% versus >50% threshold produced a PPV of 62%, an NPV of 74% and 5% indeterminate cases. When the threshold was changed to =10% vs =90%, the PPV and NPV increased to 75% and 88%, respectively, at the expense of an increase of indeterminate cases up to 62%.
CONCLUSION: When doctors assign a very high (=90%) or very low (=10%) probability of imminent death, their prognostic accuracy is improved; however, this increases the number of ‘indeterminate’ cases. This suggests that clinical predictions may continue to have a role for routine prognostication but that other approaches (such as the use of prognostic scores) may be required for those cases where doctors’ estimates are indeterminate.
BACKGROUND: The Surprise Question (SQ) "would I be surprised if this patient were to die in the next 12 months?" has been suggested to help clinicians, and especially General Practitioners (GPs), identify people who might benefit from palliative care. The prognostic accuracy of this approach is unclear and little is known about how GPs use this tool in practice. Are GPs consistent, individually and as a group? Are there international differences in the use of the tool? Does including the alternative Surprise Question ("Would I be surprised if the patient were still alive after 12 months?") alter the response? What is the impact on the treatment plan in response to the SQ? This study aims to address these questions.
METHODS: An online study will be completed by 600 (100 per country) registered GPs. They will be asked to review 20 hypothetical patient vignettes. For each vignette they will be asked to provide a response to the following four questions: (1) the SQ [Yes/No]; (2) the alternative SQ [Yes/No]; (3) the percentage probability of dying [0% no chance - 100% certain death]; and (4) the proposed treatment plan [multiple choice]. A "surprise threshold" for each participant will be calculated by comparing the responses to the SQ with the probability estimates of death. We will use linear regression to explore any differences in thresholds between countries and other clinician-related factors, such as years of experience. We will describe the actions taken by the clinicians and explore the differences between groups. We will also investigate the relationship between the alternative SQ and the other responses. Participants will receive a certificate of completion and the option to receive feedback on their performance.
DISCUSSION: This study explores the extent to which the SQ is consistently used at an individual, group, and national level. The findings of this study will help to understand the clinical value of using the SQ in routine practice.
OBJECTIVES: To identify a group of palliative care doctors who perform well on a prognostic test and to understand how they make their survival predictions.
DESIGN: Prospective observational study and two cross-sectional online studies.
SETTING: Phase I: an online prognostic test, developed from a prospective observational study of patients referred to palliative care. Phase II: an online judgement task consisting of 50 hypothetical vignettes.
PARTICIPANTS: All members of the Association of Palliative Medicine (APM) were eligible (n=~1100). 99 doctors completed the prognostic test and were included in the phase I analysis. The top 20% were invited to participate in phase II; 14/19 doctors completed the judgement task and were included in the phase II analysis.
MEASURES: Phase I: participants were asked to give a probability of death within 72 hours (0%–100%) for all 20 cases. Accuracy on the prognostic test was measured with the Brier score which was used to identify the ‘expert’ group (scale range: 0 (expert)–1 (non-expert)). Phase II: participants gave a probability of death within 72 hours (0%–100%). A mixed model regression analysis was completed using the percentage estimate as the outcome and the patient information included in the vignettes as the predictors.
RESULTS: The mean Brier score of all participants was 0.237 (95% CI 0.235 to 0.239). The mean Brier score of the ‘experts’ was 0.184 (95% CI 0.176 to 0.192). Six of the seven prognostic variables included in the hypothetical vignettes were significantly associated with clinician predictions of death. The Palliative Performance Score was identified as being the most influential in the doctors’ prognostic decision making (ß=0.48, p<0.001).
CONCLUSIONS: This study identified six clinical signs and symptoms which influenced the judgement policies of palliative care doctors. These results may be used to teach novice doctors how to improve their prognostic skills.