The Metaculus Democracy Threat Index

In recent posts on Trump and dictatorship, people have asked me - how do you know you’re not suffering from Trump Derangement Syndrome?

I take this seriously; we’ve all lost loved ones to this condition. The best check on my reasoning would be an objective measure of the health of American democracy. There are several “democracy indices” that purport to do this, but they have a mixed reputation. My impression is that most current accusations of bias are relatively weak - I agree with Claude’s analysis here - but they rely enough on “expert” opinion that I don’t expect them to convince a skeptic.

The newest entrant in this space - Metaculus Democracy Threat Index - works differently, and deserves a closer look.

Metaculus is a prediction site - like a prediction market, except that no money changes hands. People can record their guesses for how future events will turn out, which get aggregated by an algorithm (currently just a recency-weighted median, although they’ve done fanicer things in the past).

Their Democracy Threat Index is a collection of 153 questions relevant to US democracy. For example:

This question says there’s a 3.5% chance that a political party will keep an opponent off the ballot in a state election in 2026. You can see that 86 people have made forecasts about this. When probabilities like this go up on the 153 questions, the index gets higher.

This has some advantages over traditional democracy index design. It’s transparent: you can see all the questions and how they fit together. It’s crowdsourced, so there’s limited opportunity for ideologues or biased experts to put their fingers on the scale. It does a good job limiting itself to things which naturally seem democracy-related, resisting the pressure to add “and they support my preferred policy” to the definition of democracy.

But the biggest change is that Metaculus leverages modern forecasting science (popularly called “superforecasting” - although I think the people with that trademark would be unhappy to hear me use it this way). Does this help?

We care most about questions like “Did they cancel elections?” or “Did they murder protesters?” But these are coarse binary outcomes - when someone wants to know if democracy is “under threat”, they want to know when there’s increased chance of these things happening, even though they haven’t happened yet. As a forecasting platform, Metaculus can distinguish not just between “keeps elections” vs. “cancels elections”, but between 10% chance of cancelling the next elections vs. 50% chance. This both lets them focus on the big questions (rather than the small questions that are most likely to differ between one mildly-concerning regime and another) and give finer-grained estimates (a 10% vs. 50% chance of elections getting cancelled, rather than just ‘not cancelled yet’).