Haas Berkeley recently discussed new research that arose from management professor Don Moore’s “Good Judgment Project,” which trains “ordinary people to make more confident and accurate predictions over time.”
Moore and his co-authors—UPenn’s Barbara Mellers, Phillip Tetlock, Lyle Unger and Angela Minster; the University of Utah’s Elizabeth Tenney; MIT’s Heather Yang and Betterment data scientist Samuel A. Swift—combined “best practices from psychology, economics and behavioral science” within their study, “Confidence Calibration in a Multi-year Geopolitical Forecasting Competition,” which Management Science recently published.
The study “examines accuracy in forecasting over time, using a huge and unique data set.” The researchers sought out a mix of “scientists, researchers, academics and other professionals who weren’t experts in what they were forecasting.” In total, the study surveyed “494,552 forecasts by 2,860 forecasters who predicted the outcomes of hundreds of events.”
Forecasters “answered a total of 344 specific questions about geopolitical events, which targeted a specific outcome.” Over the course of a 3-year assessment, Moore says, “Our results find a remarkable balance between people’s confidence and accuracy.”
Moore adds, “What made our forecasters good was not so much that they always knew what would happen, but that they had an accurate sense of how much they knew.
Also remarkable is the way the research generated quantitative results “in a field that generally produces qualitative studies.”
The repercussions of this research could very well be exponential: “We see potential value not only in forecasting world events for intelligence agencies and governmental policy-makers, but innumerable private organizations that must make important strategic decisions based on forecasts of future states of the world.”