Jul 26, 2024 · by Joshua Tauberer
Here's the short version: We retracted the legislator rankings that we published for 2013, 2015, 2017, and 2019 including a ranking of then-Senator Kamala Harris as the most liberal senator in 2019. A newer ranking that we published in 2021 says she was the leftmost Democratic senator. Catch the difference? Right-wing media didn't like it.
We were forthright about the limitations of the analysis in 2020
Our ideology scores rank legislators according to who they partner with to co-sponsor legislation. In 2020, The Washington Post wrote about our widely reported score of then-Senator Harris:
President Trump’s reelection team must have been ecstatic to discover, buried in a wonky government-tracking website, a bit of data that helped it make the case it has wanted to make ... Why, according to GovTrack, [Biden's] pick to serve as running mate, Sen. Kamala D. Harris, is the most liberal member of the Senate! ... But that Harris isn’t generally seen as one of the most liberal members of the Senate suggests that it’s also potentially missing something, which GovTrack’s Josh Tauberer readily acknowledges.
“Obviously our analysis is at odds with what other serious analysis of Harris’s career has found, and I think the short answer is that both things can be true: She can both be sponsoring/co-sponsoring left-leaning legislation, something with typically little risk, while also being pragmatic when push comes to shove on risky and impactful decisions..." he said in an email.
And it could be politics. “Our analysis might also reflect a possible (but unsubstantiated) deliberate attempt by Harris to co-sponsor legislation to pull her record left ahead of the primaries,” he added, “or, conversely, that other senators could have been unwilling to work with Harris in order to hurt her campaign.”
It's just not possible to reduce a legislator to a perfect number. I know that some legislators even try to manipulate their own ranking on our site.
Despite my warning in The Washington Post, two months later CBS's "60 Minutes" asked Harris about our stat, and clips from the interview have been a part of Donald Trump's campaigns ever since.
What is the right way to quantify a legislator's political views?
What is the right way to quantify a legislator's political views? Should we look at what they say or how they vote? The two-axis liberal-conservative authoritarian-libertarian charts are popular, but are those the right categories? Is two dimensions enough? What fair procedure could be used to determine where a policy position (or, worse, a thousand-page bill with provisions all across the board) should fall on the chart? And how many positions should be rated: ten, a hundred, a thousand?
Our legislator ideology scores, which we began computing in 2004, followed a different path entirely. We don't look at what legislators say or how they vote. Instead, we look at the other legislators who they co-sponsor bills with. We don't use two dimensions: We only use one. We don't rate the bills they are signing onto as conservative or liberal: We don't read the bills at all. Instead, we measure how similar legislators are to each other in what set of bills they have co-sponsored. If they co-sponsor many bills together, they will get a similar score; if they co-sponsor few bills together, their scores will be far apart. Through a miracle of math, it more-or-less works. (Our methodology is described in more detail here and we have also posted its source code. Thanks to our advocacy work, Congress publishes the co-sponsorship data, so anyone can do the analysis.)
The score can be interpreted as a left—right scale measuring the dominant ideological difference or differences among Members of Congress, although of course it only takes into account a small aspect of reality, as we note on the site.
Are we doing it the right way? Well, we're doing it a way. The researchers who started VoteView.com, known in the research world as DW-NOMINATE scores, pioneered the use of math to rate legislators like this and followed a third path. Our scores are often similar to theirs but never the same. And although our procedure is fully automated, I have never called it unbiased. Our choices affect the scores. Political views are multifaceted and dynamic, giving pros and cons to any way of reducing a person's viewpoints to one or a few numbers.
It's within the margin of error
We do this anyway, despite the limitations, because we need simplification to understand the world. One of our foundational principles is to enable the American public to make data driven decisions based on the best information we can provide. It's a delicate balance to simplify, but not too much.
So it's not the only statistic we compute about ideology: We also report how often legislators join and author bipartisan bills. And we report our ideology scores over different time periods.
On our main page for Harris and similarly for every other legislator, we currently report an ideology score computed over a six-year period (in which she served the last four years). She's 4th from the left, after Sen. Sanders, Gillibrand, and Merkley. Our "report card" for the 2017-2018 session of Congress, based on a two-year period, shows the same (with Sanders and Gillibrand reversed). For the 2019-2020 session which we published after the session completed in 2021 we have Harris 2nd, after Sanders --- which we also show as 1st among the Democrats since Sanders is technically an Independent.
It fluctuates depending on how much data, how many years of co-sponsorship data, we feed the algorithm. With less data, the analysis is less reliable --- like a margin of error in polls. So what's the right amount of data to use? Is one year enough? In report card pages based on just a single year of data, we previously reported Harris ranked 8th (2017) and 1st (2019). (We only published single year statistics for odd-numbered years because in even years we published two-year statistics lined up with the congressional calendar.) The single-year 2019 statistic is what originally drew the attention of Republicans in 2020.
The jump from 8th to 1st and whether she is 1st or 2nd, it's probably all just statistical noise that doesn't reflect an underlying ground truth. Even at its most reliable, there is still a margin of error (metaphorically speaking) just from the different possible methodologies we could have used but didn't, such as using votes instead of co-sponsorship.
Standing behind our analysis
The significant fluctuation and the attention given to a cherry-picked year gave me pause. I realized that the limited data available in a single year was not sufficient to create a reliable portrait of the activity of legislators, particularly given the ebbs and flows of the legislative calendar, and therefore it did not serve as a useful tool to our users and the American public. It's important to me that we put out information that we can stand behind. After 2020, based on this reflection, GovTrack stopped publishing new single-year statistics but continued the two- and six-year statistics.
When I saw earlier this week that attention was being directed to a part of our site that I had warned was not reliable, we took the single-calendar-year statistics off the site for the same reason. All of them, and for all legislators.
The difference, as I noted at the top, is whether we say Harris was the leftmost senator in 2019 or the leftmost Democrat in 2019-2020. It is the smallest of hills to die on.
But a lot of blue checkmarks on Twitter didn't like it, all the way up to the checkmarkiest of them all. After that we started getting a steady stream of vile, and sometimes racist and antisemitic, emails about it. One nicer emailer asked, "Was the page wrong when you published it, or did it only become wrong now that it makes Harris look bad?" (I'm paraphrasing a bit), to which I answered, "It was wrong then --- and I said so." A lot of complaints about erasing history. (I don't see them complaining about Trump's VP-pick deleting his old tweets critical of Trump.). Many emails and tweets have asked us to restore the statistic because our methodology was correct and the information important for voters to know while at the same time letting us know that we have lost all of our credibility. Well, both can't really be true, can they. Trump himself continues to cite our analysis (both the retracted one and one we still have up). For them, it is all obviously about whether it's convenient, not whether it's correct.
Right-wing media ran articles with ominous headlines in passive voice like "suddenly disappears" on Fox News. (They emailed me for comment, I answered all of their questions, and they still pass it off as some invisible hand at work. It was my hand, folks.) Other outlets ran stories without asking us for comment. Most conveniently left out our newer near-identical rankings. Sean Hannity's team wrote "Is anyone buying this?". Others in the far-right media have painted this as a Soviet-style partisan censorship conspiracy. Mmmhmm.
The New York Post covered it as a retraction, which was right.
