New Publication: "Toward a more credible assessment of the credibility of science by many-analyst studies"

Katrin Auspurg and Josef Brüderl discuss the relatively new meta-scientific research design of many-analyst studies.

The paper aims to promote the use of many-analyst studies by making recommendations for the best way to realize their potential.

In many-analyst studies, numerous analysts attempt to answer the same research question using the same data. The aim is to evaluate the replicability and credibility of results from large observational data. Greater variations in the results indicate uncertainties, which reduces the credibility of individual research results. However, these studies are resource-intensive and there are doubts about their potential to provide credible assessments.

The approach offers the possibility to identify uncertainties that are independent of sampling errors. Unlike a single reanalysis or multiverse studies, the definition of “defensible” research decisions is no longer in the hands of one or a few authors. When done correctly, such studies can provide a better understanding of the modeling uncertainties that are inherent in standard scientific work with observational data.

Katrin Auspurg and Josef Brüderl identify three central issues that should be addressed in every many-analyst study:

1. Different research results are not always an indication of problematic uncertainty. Therefore, it should be clearly defined which type of uncertainty is to be examined.
2. The execution should be based on established research procedures.
   
3. Appropriate methods of meta-analysis should be used.

The authors advocate following the presented guidelines for future research, as an overestimation of the uncertainty of scientific results can endanger public trust in science. For a more reliable assessment of credibility, they therefore recommend:
i) to double-check all data processing and analysis, and
ii) to use transparent reporting standards so that readers get the best possible insight into all methods and steps of data processing used.

The complete publication in the Proceedings of the National Academy of Sciences of the United States of America (PNAS) is available here.