Assessing Medical Effectiveness in Cardio- Thoracic Surgery

4 th Workshop on Efficiency and Productivity Analysis Efficiency in the Health Sector Ricardo A. S. Castro Faculdade de Engenharia, Universidade do Porto Pedro N. Oliveira Instituto de Ciências Biomédicas Abel Salazar, Universidade do Porto Maria C. S. Portela Universidade Católica Portuguesa - Porto Ana S. Camanho Faculdade de Engenharia, Universidade do Porto João Queiroz e Melo Universidade Católica Portuguesa - Porto Assessing Medical Effectiveness in Cardio- Thoracic Surgery

Assessing Medical Effectiveness in Cardio-Thoracic Surgery Introduction Hospital Activity and Medical Performance Assessment Cardio-Thoracic Surgery Assessing Effectiveness (Life) Conclusions and Prospective Work 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 2

Introduction HOBE Main Goals: Benchmarking Web Platform Contribution to Hospital Assessment Provide Public Information about Hospitals 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 4

Hospital Activity and Medical Performance Assessment Difficulties: No theoretical reference: compare and measure relative performances (Benchmarking) Hospitals are not homogeneous units To measure outcomes besides death Inputs (Management) Managerial Control Intermediate Outputs (Management); Inputs (Clinical Part) Clinical Control Outcomes (Clinical Part) N. of doctor N. of nurses Management and support staff Drugs expenditure Fixed costs Quantity of treatment hours, nurse support, Quantity of tests, procedures Average time in a service Quantity of pacients with satisfactory outcome, by severity group Quantity of complications or readmissions Source: Chilingerian, J.A. e H.D. Sherman. 2004. Chapter 13 in Handbook on Data Envelopment Analysis. Page 496 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 6

Hospital Activity and Medical Performance Assessment The scope Inputs Clinical Control Outcomes Input 1 Input 2 Effectiveness Measure 1 Effectiveness Measure 2 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 7

Cardio-Thoracic Surgery Surgeons that treat diseases of the chest such as: Coronary artery disease; Cancer in the different chest organs; Abnormalities of the great vessels and heart valves; Birth defects of the chest and heart; Transplantation of the heart and lungs. 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 9

Cardio-Thoracic Surgery The unit under analysis Inputs (Possible Variables) Surgeon Outcomes (Dimensions to be Assessed) Internment Length of Stay (Regular and ICU) Inpatients Life Complications Patient Satisfaction Process 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 10

Assessing Effectiveness (Life) Possibilities: Mortality rate (what is usually used in benchmarking processes) Different doctors deal with different risks Clustering by severity reduces sample dimension ROC Curves Surgeon Reliability (with a measure of the risk range a surgeon is subject to) 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 12

Assessing Effectiveness (Life) ROC Curves Prediction Event Test Cases (Truth) (Prediction) 1 1 0,20 2 0 0,10 3 0 0,05 4 0 0,30 5 0 0,40 6 1 0,50 7 0 0,35 8 1 0,45 9 0 0,20 10 0 0,10 Truth Cut-off 1 0 1 TP FN 0 FP TN Cut-off 1 0 1 0 Sensitivity (TPF) α (FPF) β (FNF) Specificity (TNF) 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 13

Assessing Effectiveness (Life) ROC Curves Event Test Cases (Truth) (Prediction) 1 1 0,20 2 0 0,10 3 0 0,05 4 0 0,30 5 0 0,40 6 1 0,50 7 0 0,35 8 1 0,45 9 0 0,20 10 0 0,10 0,3 P1 P0 T1 0,67 0,33 T0 0,29 0,71 0,4 P1 P0 T1 0,67 0,33 T0 0,00 1,00 TPF 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0,5 P1 P0 T1 0,00 1,00 T0 0,00 1,00 0,2 P1 P0 T1 2 1 T0 3 4 0,2 P1 P0 T1 0,67 0,33 T0 0,43 0,57 0,1 P1 P0 T1 1,00 0,00 T0 0,57 0,43 0 P1 P0 T1 1,00 0,00 T0 1,00 0,00 0 0.2 0.4 0.6 0.8 1 FPF 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 14

Assessing Effectiveness (Life) ROC Curve True positive rate (Sensitivity) AUC 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 False negative rate (1 - Specificity) Standard error Lower bound (95%) Upper bound (95%) EuroSCORE Surgeon Surgeries (Deaths) Average Median Minimum Maximum Range A 130 (13) 0,057 0,031 0,009 0,757 0,748 B 154 (11) 0,060 0,033 0,009 0,793 0,784 C 45 (3) 0,050 0,034 0,009 0,177 0,168 D 28 (1) 0,035 0,028 0,009 0,147 0,138 E 126 (6) 0,059 0,032 0,009 0,494 0,485 F 120 (3) 0,060 0,037 0,009 0,580 0,571 G 1 (0) 0,015 0,015 0,015 0,015 0,000 H 104 (1) 0,060 0,040 0,009 0,373 0,365 I 14 (0) 0,035 0,027 0,009 0,086 0,078 J 110 (3) 0,056 0,035 0,009 0,630 0,621 K 17 (0) 0,038 0,029 0,015 0,097 0,082 L 24 (0) 0,037 0,026 0,010 0,124 0,113 Total 873 (41) 0,056 0,033 0,009 0,793 0,784 0,703 0,042 0,622 0,785 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 15

Assessing Effectiveness (Life) ROC Curve 1 0.9 True positive rate (Sensitivity) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 A B C D E F H J Surgeon AUC A 0,799 B 0,641 C 0,937 D 1,000 E 0,743 F 0,712 H 0,456 J 0,262 0 0 0.2 0.4 0.6 0.8 1 False negative rate (1 - Specificity) 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 16

Assessing Effectiveness (Life) ROC Curve Frequency 100% 80% 60% 40% D0 20% D1 0% EuroSCORE TPF (Sensitivity) 1 0.8 0.6 0.4 0.2 0 D - AUC=1 0 0.5 1 FPF (1 - Specificity) 100% J - AUC=0,262 Frequency 80% 60% 40% 20% 0% EuroSCORE J0 J1 TPF (Sensitivity) 1 0.8 0.6 0.4 0.2 0 0 0.5 1 FPF (1 - Specificity) 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 17

Assessing Effectiveness (Life) ROC Curve Frequency 70% 60% 50% 40% 30% 20% 10% 0% 13 deaths in 130 surgeries EuroSCORE A0 A1 TPF (Sensitivity) 1 0.8 0.6 0.4 0.2 0 A - AUC=0,799 0 0.5 1 FPF (1 - Specificity) Frequency 70% 60% 50% 40% 30% 20% 10% 0% 6 deaths in 126 surgeries EuroSCORE E0 E1 TPF (Sensitivity) 1 0.8 0.6 0.4 0.2 0 E - AUC=0,743 0 0.5 1 FPF (1 - Specificity) 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 18

Assessing Effectiveness (Life) Range of Risk Exposure: EuroScore Average in Alive Cases 0.22 0.20 Average EuroSCORE fo or Death Cases 0.18 0.16 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.00 0.02 0.04 0.06 0.08 Average EuroSCORE for Alive Cases A B C D E F H J 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 19

Assessing Effectiveness (Life) Metrics to Assess Life Dimension Inputs (Possible Variables) Surgeon Outcomes (2 Possible Variables) Internment Length of Stay (Regular and ICU) Inpatients Life Reliability (AUC) Risk Exposure (Average Alive s EuroSCORE) Complications Patient Satisfaction Process 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 20

Assessing Effectiveness (Life) The problem with predictors Frequency Frequency 100.00% 80.00% 60.00% 40.00% 20.00% 0.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Frequency by group: Deaths and Alives separated 0.03 0.10 0 0.17 0.23 0.30 0 0.37 0.43 0.50 0 0.57 0.63 0.70 0 0.77 7 0.83 0.90 0 0.97 EuroSCORE Frequency by number of surgeries 0.03 0.10 0.17 0.23 0.30 0.37 0.43 0.50 0.57 0.63 0.70 0.77 0.83 0.90 0.97 EuroSCORE J0 J1 J0 J1 Observed Classification Table a Alive Predicted Dead Percentage Correct Alive 829 2 99,8 Dead 40 2 4,8 Overall Percentage 95,2 a The cut value is,500 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 21

Conclusions and Prospective Work The most drastic outcome of surgeons activity can be measured by two variables: Reliability (Area Under ROC Curve) Risk Exposure (Average Risk Factor for the Alives) Prospective Work Develop adequate measure to assess risk of death Confirm proposed methodology Develop measure to assess risk of complications Define patient satisfaction and process metrics 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 23

Assessing Medical Effectiveness in Cardio-Thoracic Surgery Ricardo A. S. Castro - ricardo.alves.castro@fe.up.pt Faculdade de Engenharia, Universidade do Porto Pedro N. Oliveira Instituto de Ciências Biomédicas Abel Salazar, Universidade do Porto Maria C. S. Portela Universidade Católica Portuguesa - Porto Ana S. Camanho Faculdade de Engenharia, Universidade do Porto João Queiroz e Melo Universidade Católica Portuguesa - Porto 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 24

Extra 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 25

DEA Score with 2 Outputs and no Input Average EuroS SCORE for the Liv ving 0.0700 0.0600 0.0500 0.0400 0.0300 0.0200 0.0100 0.0000 0.000 0.200 0.400 0.600 0.800 1.000 1.200 AUC A B C D E F H J Surgeon Efficiency Super Efficiency A 114,48% 114,48% B 103,03% 103,03% C 100,00% 94,97% D 100,00% 93,65% E 101,68% 101,68% F 100,00% 95,74% H 100,00% 98,87% J 105,53% 105,53% 05-11-2012 Assessing Medical Effectiveness in Cardio-Thoracic Surgery 26