Session 2B From understanding perspectives to informing public policy the potential and challenges for Q findings to inform survey design Paper #3 Five Q-to-survey approaches: did they work? Job van Exel
Aim For each approach: Explain basic scoring rules Present descriptive statistics Compare results of approaches Discussion points Correlations between original factors: -0.05 between factors 1 and 2 0.68 between factors 1 and 3 0.09 between factors 2 and 3
Data collection n=2,714 n=228 n=1,496 n=492 Main questionnaire Version 1 1A 2 3 Main topic Q2S approaches Q2S approaches (open version) Policy Choices Social Value Orientations Part A Introductory Animation (http://www.gcu.ac.uk/endoflife/onlinesurvey/introductoryanimation/) Part B Q2S approach 1 One approach selected randomly from Q2S Part C Two approaches selected randomly from Q2S Two approaches selected randomly from Q2S approaches 2, 3, 4 and 5. Policy Choices 1 Part D approaches 2, 3, 4 and 5. approaches 2, 3, 4 and 5. (NATIONAL) Social Value Orientations Policy Choices 2 (BOARD) Part E Socio-demographics Follow up questionnaires (within 1 week of completion of main questionnaire) Full Q Sort 100 respondents sampled to reflect factor membership based on Q2S approach 1 200 respondents selected randomly from respondents answering version 1 of the main questionnaire Test Re-Test completed same questionnaire once again. This version included some difficulty questions related to first Q2S approach seen in part C.
Approach Q2S1 18 statements, 6 distinguishing each factor Scored from 1 (for completely disagree ) to 7 (for completely agree ) Factor scores calculated by summing 6 statement scores per factor Factor scores theoretically range from 6 to 42 Respondents assigned to factor with highest score; ties excluded
Results approach Q2S1 (n=4,909)
Approach Q2S2 18 statements, 6 distinguishing each factor 6 blocks, 1 statement per factor in each block Ranked from 1 to 3; 4 blocks on agreement, 2 blocks on disagreement Factor scores calculated by summing 6 statement scores per factor Factor scores theoretically range from 6 to 18 Respondents assigned to factor with lowest score; ties excluded
Results approach Q2S2 (n=2,390)
Approach Q2S3 Three abbreviated factor descriptions Scored from 1 (for very unlike my point of view ) to 7 (for very much like my point of view ) Follow-up question in case of tie in highest score Factor scores theoretically ranged from 1 to 7 Respondents assigned to factor with highest score; ties solved using follow-up question
Results approach Q2S3 (n=2,374)
Approach Q2S4 We should support an individual 2 patient's choice for treatments that give short life extensions 18 statements, 6 distinguishing each factor If a life-extending treatment for terminally ill patients is expensive, but the only treatment available, it should still be provided. I wouldn t want my life to be extended just for the sake of it - just keeping breathing is not life Sorting grid 1-1-2-3-4-3-2-1-1, ranging from 17 most disagree to most agree Statements scored by column, from -4 to +4 Factor scores calculated by summing 6 Treatments should be directed 3 statement scores per factor 41 To extend life in a way that is towards people who have a greater Factor scores theoretically range from -24 to +24 chance of survival Respondents assigned to factor with highest score; ties excluded beneficial to the patient is morally the right thing to do 27
Results approach Q2S4 (n=683)
Approach Q2S5 13 pairs of statements, 5 distinguishing each factor Statements scored by row, with score 1 if statement representing factor selected; else 0 Factor scores calculated by summing 5 statement scores per factor Factor scores theoretically range from 0 to 5 Respondents assigned to factor with highest score; ties excluded
Results of approach Q2S5 (n=2,323)
Comparison of results: assignment Factor membership according to Mean Range Q2S1 Q2S2 Q2S3 Q2S4 Q2S5 Min Max Factor 1 36.8% 38.2% 40.5% 40.4% 34.9% 38.2% 34.9% 40.5% Factor 2 49.2% 42.8% 37.6% 43.2% 40.5% 42.7% 37.6% 49.2% Factor 3 9.3% 9.0% 21.9% 11.3% 8.2% 11.9% 8.2% 21.9% Unmatched 4.7% 10.1% 0.0% 5.1% 16.4% 7.3% 0.0% 16.4%
Comparison of results: factor scores Correlations between standardized (mean=0; SD=1) factor scores on five Q2S approaches: Factor 1: 0.40 (Q2S2 vs. Q2S5) to 0.69 (Q2S1 vs. Q2S4) Factor 2: 0.62 (Q2S3 vs. Q2S4) to 0.86 (Q2S2 vs. Q2S4) Factor 3: 0.05 (Q2S4 vs. Q2S5) to 0.72 (Q2S1 vs. Q2S4)
Data collection Main questionnaire Version 1 1A 2 3 Main topic Q2S approaches Q2S approaches (open version) Policy Choices Social Value Orientations Part A Introductory Animation (http://www.gcu.ac.uk/endoflife/onlinesurvey/introductoryanimation/) Part B Q2S approach 1 One approach selected randomly from Q2S Part C Two approaches selected randomly from Q2S Two approaches selected randomly from Q2S approaches 2, 3, 4 and 5. Policy Choices 1 Part D approaches 2, 3, 4 and 5. approaches 2, 3, 4 and 5. (NATIONAL) Social Value Orientations Policy Choices 2 (BOARD) Part E Socio-demographics Follow up questionnaires (within 1 week of completion of main questionnaire) Full Q Sort 100 respondents sampled to reflect factor membership based on Q2S approach 1 200 respondents selected randomly from respondents answering version 1 of the main questionnaire Test Re-Test completed same questionnaire once again. This version included some difficulty questions related to first Q2S approach seen in part C.
Comparison with full Q sorts (n=122) Correlations between full Q sorts (i.e., their correlations with original factors) and standardized (mean=0; SD=1) factor scores on Q2S: Factor 1: n.s. (with Q2S5; n=19), 0.73 (with Q2S2 & Q2S4) and 0.78 (with Q2S1 & Q2S3) Factor 2: 0.77 (with Q2S4), 0.80 (with Q2S3), 0.82 (with Q2S1), 0.87 (with Q2S2) and 0.90 (with Q2S5; n=36) Factor 3: n.s. (with Q2S5; n=19), 0.60 (with Q2S4), 0.74 (with Q2S1 & Q2S3) and 0.77 (with Q2S2) Correlations between original factors: -0.05 between factors 1 and 2 0.68 between factors 1 and 3 0.09 between factors 2 and 3
Future work Explore more advanced scoring rules Brown s factor index score (Q2S approach 1) Latent Class Analysis (Q2S approach 1) Q factor analysis (Q2S approach 4) Associate factor scores with Personal characteristics Attitudinal questions Policy questions
Discussion points Among these five, is there a (theoretically) Scores preferred from 5 (for a method little for including the results of a Q study in a survey? like my point of view ) to 7 (for very much like my point of view ) What to do with respondents scoring below a certain threshold? And what does that mean? Suggestions for analysis? Alternative methods?