What do we know about the disruption index in scientometrics?

The disruption index (DI₁) is a bibliometric index aimed at quantifying groundbreaking re-search. The DI₁ has seen widespread application since its introduction by Funk and Owen-Smith (2017) and researchers have proposed numerous modified variants of the index. The DI₁ recently received much media attention because a study published in Nature found that papers in all disciplines and patents are becoming less disruptive over time. Park et al. (2023) report that the absolute number of scientific breakthroughs has remained constant since the end of World War II in spite of the massive growth of the global science system in the past decades. In other words, this result implies that the exponential expansion of science in recent decades has not led to an equally substantial harvest of original ideas and groundbreaking inventions. The article made waves in and beyond the science system and prompted a public debate surrounding the question of if and why science seems to be running out of steam.

While the finding that both patents and papers are getting »less bang per buck« is certainly spectacular, it is important not to jump unreflectively and straight forward to far reaching conclusions (science policy actions). Park et al. (2023, p. 143) themselves point out that the DI₁ “is a relatively new index of innovative activity and will benefit from future work on its behaviour and properties”. Therefore, any meaningful discussion about the results of Park et al. (2023) (as well as the results of any other study involving the DI₁) requires a detailed un-derstanding of the index’s properties and limitations, which have been studied in several (em-pirical) studies since 2019. In order to provide detailed insights into the properties and limita-tions of (different variants of) the DI₁, this review paper provides a systematic review of the current literature on DI₁ and its modified variants. The review consists of four parts. In the first part, the technical and theoretical properties of the DI₁ are explained. The second part covers the numerous modified index variants of the DI₁ researchers have proposed so far. The third part highlights possible disadvantages of using citation data to measure disruption. The fourth part provides an overview of the literature on the validity of disruption scores. This part discusses the studies dealing with the important question whether the indices measure what they propose to measure.

Calculation of DI₁
 

Main results

1. Limitations of the DI₁:

The DI1 has several limitations that have not been addressed adequately by empirical studies. Highly cited publications, time-sensitivite biases as well as insufficient coverage of data in bibliometric databases may lead to unreliable and even misleading results. In addition to all of this, the index is also inconsistent. In summary, it seems that the DI1 is not suitable for historic analysis.

2. Validity:

Studies that test whether the DI₁ (and its variants) are able to measure what they propose to measure lead to mixed results. Differences in sample construction and statistical procedures further complicate the interpretation of the mixed results. How-ever, the literature does offer one finding that is inconsistent across all studies: Some modified variants of the DI1 offer considerable improvements compared to the DI₁.

3. Best practice:

Best practice guidelines for the application of the DI₁ and its variants are provided in the discussion section of the review article. Researchers may tackle some of the weaknesses of disruption indices by a) informing about the variety of disruption indices and choosing the variant that is appropriate for the research question; b) choosing an adequate citation window; and c) paying attention to proper data prepa-ration.

A comprehensive review of the literature shows that more research on the validity of disrup-tion scores as well as a more precise understanding of disruption as a theoretical construct is needed before disruption indices can be used in the research evaluation practice. Before we know about the disruption index and its limitations it is not advisable to use studies based on disruption indices as groundwork for science policy actions.

Literature

Funk, R. J., & Owen-Smith, J. (2017). A dynamic network measure of technological change.
Management Science, 63(3), 791-817. https://doi.org/10.1287/mnsc.2015.2366
Park, M., Leahey, E., & Funk, R. J. (2023). Papers and patents are becoming less disruptive over time.
Nature, 613(7942), 138-144. https://doi.org/10.1038/s41586-022-05543-x