As described in Section 3.3.2, the panelists were asked to supply three numbers for each question: a MV, a LB, and a UB. The MV has a nominal 50/50 chance of falling above or below the correct value. The interval (LB, UB) has a nominal 90% chance of covering the correct value.

The following tables summarize the responses made in the training exercise. There were between 15 and 17 sets of responses to each question. (Although there were only 12 panelists on the panel, members of the facilitation team were also invited to participate.) The number of respondents is indicated following each question. The table columns summarize the responses relative to the CV. The first column indicates the number of respondents where CV < LB, i. e., where the coverage interval fell above the CV; the third column indicates the number of respondents where CV > UB, i. e., where the coverage interval fell below the CV. Thus, the total of the first and third columns is the number of respondents whose coverage intervals did not cover the CV. The second column lists three numbers that summarize the set of MVs provided for each row of the table. These are the lower quartile (LQ), median and upper quartile (UQ), respectively. About one quarter of the MVs are less than the LQ and about one quarter of the MVs are greater than the UQ. Hence the interquartile interval (LQ, UQ), denoted by IQI, contains about one half of the MVs. (These three summary statistics are used to construct box and whisker plots, as described in Appendix L.) For ease of reference, the rounded correct values are listed following the conditions for the Q2 — Q4 tables.

Q1. According to the 2000 census, how many men 65 or over were there in the U. S.? (N = 17)

(CV = 14.4 million)

Respondents tended to overestimate the CV. Since LQ = 16, about three quarters of the MVs were larger than the CV. However, percent coverage at 82% was near the nominal 90%, with 3 (18%) lying above the CV and none lying below.

Q2. How many American men age 65 or older suffered from the following chronic conditions in 1995? (N = 15)

Table C.3 Summary of Respondent Results for Question Q2

Four of the six IQIs covered the CV, and the two which did not almost did. Three of the medians were above the CV and three were below. Thus, the MVs for the six conditions as a whole exhibited no systematic bias in estimating the CVs. However, the coverage intervals tended to underestimate the CVs. Of the 90 coverage intervals, 27 (30%) lay below the CV and 8 (9%) lay above. The average percent coverage of all 90 intervals was 61%. Over the six conditions, the percent coverage ranged from a low of 33% to a high of 80%.

Q3. What is the ratio of the rate for men 45- 64 years old to the rate for men 65 and older for each of the conditions listed? (N = 16)

Table C.4 Summary of Respondent Results for Question Q3

Five of the six IQIs covered the CV, but respondents tended to underestimate the CV. Five of the six medians were below the CV. Of the 96 coverage intervals, 22 (23%) lay below the CV and 5 (5%) lay above. The average percent coverage of all 96 intervals was 72%. Over the six conditions, the percent coverage ranged between 62% and 88%.

Q4. What is the ratio of the rate for men under 45 years old to the rate for men 45 — 64 years old for each of the conditions listed? (N = 16)

Table C.5 Summary of Respondent Results for Question Q4

Five of the six IQIs covered the CV, but respondents tended to overestimate the CV. Four of the six medians were above the CV, and two were equal or almost equal to the CV. Of the 96 coverage intervals, 20 (21%) lay above the CV and 8 (8%) lay below. The average percent coverage of all 96 intervals was 71%. Over the six conditions, the percent coverage ranged between 50% and 88%.

C. 3 Discussion

The results of the training exercise were consistent with several of the basic premises underlying the elicitation structure and methodology. First, apart from Q1, the responses to the other three questions as a whole did not exhibit any systematic over — or under — estimation bias. Q2 had no systematic bias, Q3 tended to underestimate, and Q4 tended to overestimate the CVs. This result is consistent with the basic premise of the elicitation process, which is that the panel responses as a whole have no systematic bias (see Section 3.3).

Second, the percent coverage of the (LB, UB) intervals were less than the nominal 90% for all four questions. Q1 had the highest percent coverage at 82%, perhaps because the question dealt with demographic data with which the respondents were relatively more familiar. Q3 and Q4 had the next highest percent coverage at about 71% each and Q2 had the lowest percent coverage at 61%. This result is consistent with the rationale for the overconfidence adjustments made to the panelists’ uncertainty intervals (see Section 5.6.2).

Third, the two questions (Q3 and Q4) that asked about ratios of rates had higher percent coverage than the question (Q2) that asked about absolute rates. This result is consistent with the rationale for the basic structure of the elicitation questions, which ask about relative rather than absolute LOCA frequencies (see Section 3.8).