I was listening to the Risky Business podcast, in which Maria Konnikova and Nate Silver discuss why you should put your probabilities in numbers rather than words. I wanted to add that I think it’s even worse than they suggest, because of variable consequences of the event under discussion.
The segment is introduced via an anecdote the general gist of which is that an American analyst in the early 1950s reported on the possibility of a Soviet invasion of Yugoslavia using the words “significant possibility”, which he himself understood to mean around 65%. Unfortunately it was read to mean around 25%. So the anecdote serves as a cautionary tale with the lesson being that one should communicate probabilities in numbers, not descriptive words.
There follows an interesting game you can play with anyone, simply describe a probability in words and then see if you agree on the actual number it means. In the podcast Nate and Maria disagree on the very first turn with Nate ascribing 95% to “highly probable”, whilst Maria needs only 90% for that.
I think the situation here is even worse than described. I think you commonly disagree with yourself regarding the numerical probability implied by a descriptive term. Meaning that you do not apply these terms consistently. I think whenever you use such terms, they are anchored according to some base rate for the event in question, and also affected by the consequences of the event in question. So for example if two sports teams A and B are meeting in a match, I may predict that it is “highly unlikely” that team B wins. But what I probably mean by that is something like 5%. That’s because upsets in sports happen all the time, and the consquences are rarely dire.
However, if I were to say that I think it is “highly unlikely” that Britain is forced to utilise its nuclear arsenal in a conflict this decade, I probably have a much lower probability that 5% in mind. Something less than 1%. That, I think, is because both the base rate for nuclear weapon usage in conclicts is low and because the consequences would clearly be significantly greater.