Article InformationEdit

Title: Evaluating Subjective Expert Judgment for Science Policy: An Application of the Bayesian Truth Serum

Author(s): Rebecca Weiss

Affiliation: Massachusetts Institute of Technology

Presented At: STiS 2008

Primary Topic Area: Science and the Public


Subjective expert judgments are essential to creating good policies; this is especially true in science and technology policy. However, it is not easy to evaluate subjective judgments in a way that accurately reflects both the honest opinion of experts as well as objective truth. Simple measures like the averaged judgment, or majority rule, can be inadequate since such information does not account for variations in the quality of expert opinion. This may occur when some judgments are unpopular or unusual.

The “Bayesian Truth Serum” or BTS (Prelec, 2004) is a scoring method that gives experts incentives for truthfully reporting judgments in settings where no immediate objective truth criterion is available. This is the case with long-range scientific forecasts. According to theory, the method is also able to identify the judgment with the highest probability of being objectively true provided that the participating experts are rational Bayesian decision-makers and that a single right answer exists (i.e., the method optimally aggregates experts’ information).

This project aims to determine whether BTS can aid the science policy-making process, where there are several potential “right” answers but where recognized experts would establish one as being more truthful than others. A selection of recent scientific studies with potential impact for science policy will be presented to recognized experts. Their judgments will be assessed using BTS and compared with other statistical measures to determine whether BTS is more accurate at estimating both objective truth and the truthfulness of expert judgment.

Ad blocker interference detected!

Wikia is a free-to-use site that makes money from advertising. We have a modified experience for viewers using ad blockers

Wikia is not accessible if you’ve made further modifications. Remove the custom ad blocker rule(s) and the page will load as expected.