A Times Higher Education piece making the rounds last week found that most published philosophy papers are never cited. More exactly, of the studied philosophy papers published in 2012, more than half had no citations indexed in Web of Science five years later.

At Daily Nous, the discussion of that finding turned up some interesting follow-up questions and findings. In particular, Brian Weatherson found quite different figures for papers published in prestigious philosophy journals. In the journals he looked at, 89% of the papers published in 2012 had at least one citation in Web of Science five years later.¹ And more than half had five or more citations.

That’s a pretty striking difference: >50% vs. ~11%! Seems like where you publish your paper makes a big difference to your chances of going uncited.

Shocking, I know.

But this got me thinking about Kieran Healy’s analysis from a few years back. He found an “uncitation” rate higher than Weatherson’s 11%—almost 20%—even though he was looking at just four of philosophy’s most prominent journals: Journal of Philosophy, Mind, Noûs, and Philosophical Review. (He found that around half of the papers in these journals had five citations or fewer.)

So I wondered: what’s with the discrepancy? Do these journals not necessarily get the most citations? Or is it that Healy was looking at papers from 1993 to 2013, and things changed somehow over those two decades, so that papers published in 2012 tend to get discussed more than papers from 1993. Or is it just a symptom of when Healy collected his data? Papers published in, say, 2011 wouldn’t have had much time to gather citations by 2013 when Healy (apparently?) gathered his data.

Let’s take a look.

The Healy 2100

Since I don’t have Healy’s raw data, I went to Web of Science and grabbed their data for all papers published over 1993–2013 in the “Healy 4” journals. I ended up with a couple hundred more papers than Healy looked at—not sure why. But the list of 2,100 papers he studied is available on GitHub. So I focused on just those to get more of an apples-to-apples comparison.

Then I tried to reproduce his original findings, especially this histogram. Here’s my version:

I got pretty close, but I didn’t manage to reproduce his results exactly. I found about 19.9% of the papers had no citations by the end of 2013, compared to Healy’s ~18.5%. And I found about 58.6% with five or fewer citations, compared with Healy’s “just over half”.

Still, the match is pretty close, so let’s go on to see how these papers have aged since 2013.

Where Are They Now?

If we include citations up to the present day, only about 9.7% of these 2,100 papers have no citations, and about 39.1% have five or fewer. Here’s the updated histogram:

So it looks like the discrepancy with Weatherson’s result is (partly?) down to the obvious thing. There hadn’t been enough time for the later papers in Healy’s data set to accrue citations.

Cutting Out Supplements

A lot of these 2,100 papers are actually from the two supplements to Noûs: Philosophical Issues and Philosophical Perspectives. And it turns out they’re making a big difference.

Here’s what things look like when we cut supplements out:

Now only 3.8% of our 1,677 papers have no citations to date, and just 30.9% have five or fewer.

Sliding Windows

We’ve been looking at all citations accumulated to date for these papers, which for older papers means 25 years’ worth of opportunity for discussion. For more direct comparison to the THE analysis mentioned at the outset, we can look at just the five-year window following each paper’s publication.

So, how many citations did these papers accrue just within five years of being published? Looking at only the “core” papers again (no supplements), 14.4% had no citations within five years of publication, and 69.6% had five or fewer.

That’s still a bit higher than the 11% (respectively 50%) found by Weatherson. So we have to ask: are things changing? Are recent papers in these journals accruing citations faster?

It seems so. Here are the “uncitation” rates for the years 1993–2013:

And here are the “five or fewer citations” rates:

Update: Eric Schliesser pointed out to me what should have been obvious here, namely the internet. When Google took over the search engine industry around the year 2000, there was a citation boom across the board. So you’d naturally expect 2012 to be a very different year than 1993 as far as uncitation goes.

At first I thought the data plainly vindicated the “Google effect” hypothesis, and I had some plots up here to show as much. But it turned out Web of Science had snuck a bunch of supplemental Phil Issues/Phil Perspectives papers past me unmarked!

With those removed, the Google effect isn’t looking so big. Here’s the long view, from 1970 to 2013:

Apparently papers with zero citations after five years have been declining for a while now. There does still seem to be a significant Google effect in the “five or fewer” measure:

But it still looks like there were changes before Google. Does this reflect a growing profession? A faster-moving profession? An increasing number of publications? Longer bibliographies? Just idiosyncracies of Web of Science’s indexing? Something else? I’m not sure.