Project sheet
CALL: PRIN: RESEARCH PROJECTS OF RELEVANT NATIONAL INTEREST - Call for proposals 2022
TITLE: Measuring Scientific Output via Preferences Elicitation
DURATION: 24 months from 09/28/2023 to 09/27/2025
RESPONSIBLE: Prof. Magnani Marco (Head of Unit)
ERC (European Research Council) SECTOR:SH 1 - "Individuals, Markets and Organisations"
Sustainable Development Goals (SDGs): Goal 8: Decent work and economic growth
Website: PeerRank - Researchers rank it better
Brief description
The evaluation of scientific production is increasingly based on citation data, despite the fact that this has never been their raison d'être; this has generated and still generates strategic behavior and frequent controversy. Despite this, the use of such metrics for routine evaluations of large quantities of publications is often unavoidable. The strong limitations of this approach, and in particular the lack of consensus on which method to use to evaluate scientific output, hinder the evaluation of science and thus its development-in societies where knowledge plays an increasingly central and critical role. This project aims to improve the evaluation of scientific journals by addressing three main issues:
1) Whether or not there is a consensus among scientists on which journals are most relevant.
2) Which existing metrics best reflect the scientific community's judgment of journal quality.
3) The aggregation of the community consensus into a new metric, PeerRank, that best reflects their opinion.
The project aims to collect scientists' preferences through a "smart poll," combining bottom-down approaches - in which each researcher expresses a ranking of certain journals - bottom-up approaches - in which instead the researcher compares existing indicators. The results will be analyzed in the context of the research evaluation debate, considering the diversity among disciplines. At the stage of aggregating the preferences expressed by the participants, the choice of an algorithm that optimally exploits the information but at the same time lends itself as little as possible to manipulation is crucial. For this reason, the project includes a theoretical study component of aggregation algorithms, as well as an empirical analysis of strategic behavior in contexts, such as the Eurovision Song Contest, in which ordinal preferences are expressed in the presence of clear conflicts of interest.
