Assessing Medical Effectiveness in Cardio- Thoracic Surgery

Transcription

1 4 th Workshop on Efficiency and Productivity Analysis Efficiency in the Health Sector Ricardo A. S. Castro Faculdade de Engenharia, Universidade do Porto Pedro N. Oliveira Instituto de Ciências Biomédicas Abel Salazar, Universidade do Porto Maria C. S. Portela Universidade Católica Portuguesa - Porto Ana S. Camanho Faculdade de Engenharia, Universidade do Porto João Queiroz e Melo Universidade Católica Portuguesa - Porto Assessing Medical Effectiveness in Cardio- Thoracic Surgery

2 Assessing Medical Effectiveness in Cardio-Thoracic Surgery Introduction Hospital Activity and Medical Performance Assessment Cardio-Thoracic Surgery Assessing Effectiveness (Life) Conclusions and Prospective Work Assessing Medical Effectiveness in Cardio-Thoracic Surgery 2

3 Assessing Medical Effectiveness in Cardio-Thoracic Surgery Introduction Hospital Activity and Medical Performance Assessment Cardio-Thoracic Surgery Assessing Effectiveness (Life) Conclusions and Prospective Work Assessing Medical Effectiveness in Cardio-Thoracic Surgery 3

4 Introduction HOBE Main Goals: Benchmarking Web Platform Contribution to Hospital Assessment Provide Public Information about Hospitals Assessing Medical Effectiveness in Cardio-Thoracic Surgery 4

5 Assessing Medical Effectiveness in Cardio-Thoracic Surgery Introduction Hospital Activity and Medical Performance Assessment Cardio-Thoracic Surgery Assessing Effectiveness (Life) Conclusions and Prospective Work Assessing Medical Effectiveness in Cardio-Thoracic Surgery 5

6 Hospital Activity and Medical Performance Assessment Difficulties: No theoretical reference: compare and measure relative performances (Benchmarking) Hospitals are not homogeneous units To measure outcomes besides death Inputs (Management) Managerial Control Intermediate Outputs (Management); Inputs (Clinical Part) Clinical Control Outcomes (Clinical Part) N. of doctor N. of nurses Management and support staff Drugs expenditure Fixed costs Quantity of treatment hours, nurse support, Quantity of tests, procedures Average time in a service Quantity of pacients with satisfactory outcome, by severity group Quantity of complications or readmissions Source: Chilingerian, J.A. e H.D. Sherman Chapter 13 in Handbook on Data Envelopment Analysis. Page Assessing Medical Effectiveness in Cardio-Thoracic Surgery 6

7 Hospital Activity and Medical Performance Assessment The scope Inputs Clinical Control Outcomes Input 1 Input 2 Effectiveness Measure 1 Effectiveness Measure Assessing Medical Effectiveness in Cardio-Thoracic Surgery 7

8 Assessing Medical Effectiveness in Cardio-Thoracic Surgery Introduction Hospital Activity and Medical Performance Assessment Cardio-Thoracic Surgery Assessing Effectiveness (Life) Conclusions and Prospective Work Assessing Medical Effectiveness in Cardio-Thoracic Surgery 8

9 Cardio-Thoracic Surgery Surgeons that treat diseases of the chest such as: Coronary artery disease; Cancer in the different chest organs; Abnormalities of the great vessels and heart valves; Birth defects of the chest and heart; Transplantation of the heart and lungs Assessing Medical Effectiveness in Cardio-Thoracic Surgery 9

10 Cardio-Thoracic Surgery The unit under analysis Inputs (Possible Variables) Surgeon Outcomes (Dimensions to be Assessed) Internment Length of Stay (Regular and ICU) Inpatients Life Complications Patient Satisfaction Process Assessing Medical Effectiveness in Cardio-Thoracic Surgery 10

11 Assessing Medical Effectiveness in Cardio-Thoracic Surgery Introduction Hospital Activity and Medical Performance Assessment Cardio-Thoracic Surgery Assessing Effectiveness (Life) Conclusions and Prospective Work Assessing Medical Effectiveness in Cardio-Thoracic Surgery 11

12 Assessing Effectiveness (Life) Possibilities: Mortality rate (what is usually used in benchmarking processes) Different doctors deal with different risks Clustering by severity reduces sample dimension ROC Curves Surgeon Reliability (with a measure of the risk range a surgeon is subject to) Assessing Medical Effectiveness in Cardio-Thoracic Surgery 12

13 Assessing Effectiveness (Life) ROC Curves Prediction Event Test Cases (Truth) (Prediction) 1 1 0, , , , , , , , , ,10 Truth Cut-off TP FN 0 FP TN Cut-off Sensitivity (TPF) α (FPF) β (FNF) Specificity (TNF) Assessing Medical Effectiveness in Cardio-Thoracic Surgery 13

14 Assessing Effectiveness (Life) ROC Curves Event Test Cases (Truth) (Prediction) 1 1 0, , , , , , , , , ,10 0,3 P1 P0 T1 0,67 0,33 T0 0,29 0,71 0,4 P1 P0 T1 0,67 0,33 T0 0,00 1,00 TPF ,5 P1 P0 T1 0,00 1,00 T0 0,00 1,00 0,2 P1 P0 T1 2 1 T ,2 P1 P0 T1 0,67 0,33 T0 0,43 0,57 0,1 P1 P0 T1 1,00 0,00 T0 0,57 0,43 0 P1 P0 T1 1,00 0,00 T0 1,00 0, FPF Assessing Medical Effectiveness in Cardio-Thoracic Surgery 14

15 Assessing Effectiveness (Life) ROC Curve True positive rate (Sensitivity) AUC False negative rate (1 - Specificity) Standard error Lower bound (95%) Upper bound (95%) EuroSCORE Surgeon Surgeries (Deaths) Average Median Minimum Maximum Range A 130 (13) 0,057 0,031 0,009 0,757 0,748 B 154 (11) 0,060 0,033 0,009 0,793 0,784 C 45 (3) 0,050 0,034 0,009 0,177 0,168 D 28 (1) 0,035 0,028 0,009 0,147 0,138 E 126 (6) 0,059 0,032 0,009 0,494 0,485 F 120 (3) 0,060 0,037 0,009 0,580 0,571 G 1 (0) 0,015 0,015 0,015 0,015 0,000 H 104 (1) 0,060 0,040 0,009 0,373 0,365 I 14 (0) 0,035 0,027 0,009 0,086 0,078 J 110 (3) 0,056 0,035 0,009 0,630 0,621 K 17 (0) 0,038 0,029 0,015 0,097 0,082 L 24 (0) 0,037 0,026 0,010 0,124 0,113 Total 873 (41) 0,056 0,033 0,009 0,793 0,784 0,703 0,042 0,622 0, Assessing Medical Effectiveness in Cardio-Thoracic Surgery 15

16 Assessing Effectiveness (Life) ROC Curve True positive rate (Sensitivity) A B C D E F H J Surgeon AUC A 0,799 B 0,641 C 0,937 D 1,000 E 0,743 F 0,712 H 0,456 J 0, False negative rate (1 - Specificity) Assessing Medical Effectiveness in Cardio-Thoracic Surgery 16

17 Assessing Effectiveness (Life) ROC Curve Frequency 100% 80% 60% 40% D0 20% D1 0% EuroSCORE TPF (Sensitivity) D - AUC= FPF (1 - Specificity) 100% J - AUC=0,262 Frequency 80% 60% 40% 20% 0% EuroSCORE J0 J1 TPF (Sensitivity) FPF (1 - Specificity) Assessing Medical Effectiveness in Cardio-Thoracic Surgery 17

18 Assessing Effectiveness (Life) ROC Curve Frequency 70% 60% 50% 40% 30% 20% 10% 0% 13 deaths in 130 surgeries EuroSCORE A0 A1 TPF (Sensitivity) A - AUC=0, FPF (1 - Specificity) Frequency 70% 60% 50% 40% 30% 20% 10% 0% 6 deaths in 126 surgeries EuroSCORE E0 E1 TPF (Sensitivity) E - AUC=0, FPF (1 - Specificity) Assessing Medical Effectiveness in Cardio-Thoracic Surgery 18

19 Assessing Effectiveness (Life) Range of Risk Exposure: EuroScore Average in Alive Cases Average EuroSCORE fo or Death Cases Average EuroSCORE for Alive Cases A B C D E F H J Assessing Medical Effectiveness in Cardio-Thoracic Surgery 19

20 Assessing Effectiveness (Life) Metrics to Assess Life Dimension Inputs (Possible Variables) Surgeon Outcomes (2 Possible Variables) Internment Length of Stay (Regular and ICU) Inpatients Life Reliability (AUC) Risk Exposure (Average Alive s EuroSCORE) Complications Patient Satisfaction Process Assessing Medical Effectiveness in Cardio-Thoracic Surgery 20

21 Assessing Effectiveness (Life) The problem with predictors Frequency Frequency % 80.00% 60.00% 40.00% 20.00% 0.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Frequency by group: Deaths and Alives separated EuroSCORE Frequency by number of surgeries EuroSCORE J0 J1 J0 J1 Observed Classification Table a Alive Predicted Dead Percentage Correct Alive ,8 Dead ,8 Overall Percentage 95,2 a The cut value is, Assessing Medical Effectiveness in Cardio-Thoracic Surgery 21

22 Assessing Medical Effectiveness in Cardio-Thoracic Surgery Introduction Hospital Activity and Medical Performance Assessment Cardio-Thoracic Surgery Assessing Effectiveness (Life) Conclusions and Prospective Work Assessing Medical Effectiveness in Cardio-Thoracic Surgery 22

23 Conclusions and Prospective Work The most drastic outcome of surgeons activity can be measured by two variables: Reliability (Area Under ROC Curve) Risk Exposure (Average Risk Factor for the Alives) Prospective Work Develop adequate measure to assess risk of death Confirm proposed methodology Develop measure to assess risk of complications Define patient satisfaction and process metrics Assessing Medical Effectiveness in Cardio-Thoracic Surgery 23

24 Assessing Medical Effectiveness in Cardio-Thoracic Surgery Ricardo A. S. Castro - ricardo.alves.castro@fe.up.pt Faculdade de Engenharia, Universidade do Porto Pedro N. Oliveira Instituto de Ciências Biomédicas Abel Salazar, Universidade do Porto Maria C. S. Portela Universidade Católica Portuguesa - Porto Ana S. Camanho Faculdade de Engenharia, Universidade do Porto João Queiroz e Melo Universidade Católica Portuguesa - Porto Assessing Medical Effectiveness in Cardio-Thoracic Surgery 24

25 Extra Assessing Medical Effectiveness in Cardio-Thoracic Surgery 25

26 DEA Score with 2 Outputs and no Input Average EuroS SCORE for the Liv ving AUC A B C D E F H J Surgeon Efficiency Super Efficiency A 114,48% 114,48% B 103,03% 103,03% C 100,00% 94,97% D 100,00% 93,65% E 101,68% 101,68% F 100,00% 95,74% H 100,00% 98,87% J 105,53% 105,53% Assessing Medical Effectiveness in Cardio-Thoracic Surgery 26