Quantitative Social Research
print
Search
Hinweise zur Datenübertragung bei der Google™ Suche
Language Selection
Breadcrumb Navigation
Content

Project phase 2 (2025-2028)

Subproject "Enhancing the Robustness of Observational Social Science Research by Computational Multi-Model Analyses"

The project conducts analyses on the reproducibility and robustness of articles that use the same large-N observational data (European Social Survey). However, the articles differ in disciplines, journals, author constellations, and other factors that are likely associated with varying levels of reproducibility.

Our project proceeds in four steps: During the first project phase, we (A) assessed the accessibility of replication materials for approximately 1,200 articles in an "openness audit" (contacting authors regarding the availability of data and analysis code). We then (B) conducted reproducibility analyses on a random subset of 100 articles (i.e., can the authors' results be reproduced by applying their code to their data?). For the second project phase, we plan to conduct (C) “accuracy/congruence checks” for these articles (e.g., absence of coding errors, congruence between what is reported in the article and what is actually done with the analysis code). In the final step, we focus on (D) robustness analyses (i.e., do the results remain stable under seemingly minor changes to data preparation or analysis, such as changes in weighting, imputation, or outlier treatment?).

Through these four steps, we aim to conduct a comprehensive audit of potential threats to transparent and credible research. In contrast to existing audits, we include publications with both high and low impact. Our audit also enables comparisons across disciplines with varying levels of implementation of the FAIR principles (findable, accessible, interoperable, reusable materials). This allows us to identify research areas where improvement measures appear particularly necessary and effective.

The three closely interconnected overarching research goals for the second project phase are:

  1. Finalizing the analysis of the extent of reproducible and robust results across the four audit stages (as a contribution to the META-REP “What?” question);
  2. Analyzing conditions for varying levels of reproducibility and robustness, including article-, author-, and journal-level characteristics (as a contribution to the META-REP “Why?” question); and
  3. Developing easily implementable measures (such as computer code) that can improve reproducibility and robustness (as a contribution to the META-REP “How?” question).