[ZNPaaneah]

I am reading a book by Tetlock. We are all familiar with his work, and Ohio and others on this forum have paraphrased it repeatedly, even if you are not familiar with the name. In 2006 he had a very influential book Expert Political Judgement: How Good is it? The results of that work are commonly paraphrased saying that these political forecasts by experts are as good as a monkey throwing darts. Here is an example, taken from Amazon.com, a reviewer of that book: "The definitive work on this question. . . . Tetlock systematically collected a vast number of individual forecasts about political and economic events, made by recognised experts over a period of more than 20 years. He showed that these forecasts were not very much better than making predictions by chance, and also that experts performed only slightly better than the average person who was casually informed about the subject in hand."---Gavyn Davies, Financial Times

The problem is that this is very misleading and misses the crucial point of his research which is that there are some people who are remarkably accurate forecasters. If you look at the top 10% they are reliable, accurate, and consistent. He has been working on the science of forecasting, and has shown excellent year on year improvement.

The process is very simple: create a model, forecast, evaluate the result. Based on evaluation make adjustments to the model, forecast, evaluate, etc. The more times you do this the more precise and reliable you get. This is why weather forecasts have improved so much, this is the process they follow. This is also why the pundits who complain about climate change models being inaccurate are morons. This is the process you follow. Every time you are "inaccurate" you are learning how to weigh each factor, how much more data you need, and a thousand other ways to improve the forecast. It is a process. The issue is not if the model can perfectly predict the outcome, but can it predict it more accurately than the previous model. As long as we improve from one iteration to the next we will get to that very accurate model eventually.

In science we do that, but apparently no one is measuring and grading the accuracy of political pundits. No one is holding them to account. There is no rating system for their reliability. So whereas a car is rated and compared and the results are available for all to see, no one holds these pundits accountable. That is the service that needs to take place.

What Tetlock did was encourage more than a thousand people to participate in making political forecasts each day, 9am EST. Then they would grade them. They would take the 100 or 200 burning questions of the day, and they would try to keep the forecasts between 1 month and 5 years, but mainly in the 1 month to 18 month range.

It seems to me that we could create a "good housekeeping seal of approval" for pundits. Make them do this for 1 year, grade them, and only those who score at the "expert" level get the seal.

In addition there is a science to how they go about preparing these forecasts, so there could be a graduate degree in this. For example, someone with an undergraduate degree in Economics might do a graduate degree in forecasting.