The CRM Signs the CoARA Agreement

Expanding from earlier initiatives like the San Francisco Declaration on Research Assessment (DORA) or the Leiden Manifesto, CoARA widens its scope beyond a simple statement of purpose. It actively encourages organizations to pledge their commitment to overhaul their assessment methods within a defined time frame, spanning from 2022 to 2027.

The Centre de Recerca Matemàtica (CRM) has officially signed the Agreement on Reforming Research Assessment joining the Coalition for Advancing Research Assessment (CoARA). CoARA is a global coalition that unites a diverse array of institutions engaged in research evaluation, along with their respective associations, committed to reimagining and reforming the landscape of research evaluation. This alliance, first launched in December 2022, involves research funding bodies, universities, research institutions, and national and regional assessment agencies. All these entities are motivated to cooperate, striving for a transformative overhaul grounded in shared principles and commitments outlined in the Agreement.

Research assessment has long been a hotbed of debate within the academic community, with traditional metrics such as journal impact factors and citation count often falling short of capturing the full spectrum of research quality. The agreement, structured around ten core commitments, suggests acknowledging the wide array of practices, activities, and professional paths within the realm of research. It also advocates for the incorporation of qualitative benchmarks to reduce the reliance on metrics tied to journals and publications.

Furthermore, the agreement calls for the allocation of ample resources for research assessment, along with the examination and enhancement of novel evaluation criteria and procedures. This new approach also places an emphasis on qualitative indicators and metrics, ensuring that the evaluation process is sensitive to the broader societal context in which research operates.

The organizations that have endorsed the CoARA agreement aim to revise and enhance the standards, tools, and procedures used to assess scientific endeavours, while also increasing awareness of and providing training to their staff on this reform initiative. Members of the coalition are committed to sharing their best practices and experiences to facilitate mutual learning, keeping each other informed about their progress in adhering to the principles and fulfilling their commitments, and conducting evaluations of assessment practices and criteria based on robust evidence related to research evaluation.

A common path towards advancing assessment methods for research.

In the pursuit of fair and responsible research assessment, the CoARA agreement commits to recognizing the diversity of research contributions and careers, aligning them with research needs. Among other aspects, the core principles of the agreement recommend assessment primarily relying on qualitative peer reviews, complemented by responsible quantitative indicators. CoARA also encourages leaving behind inappropriate metrics like the Journal Impact Factor and rankings of research organizations, and pledges to allocate resources for the necessary changes in research assessment. Processes should also be reviewed and refined, and transparent communication and guidance should be promoted to raise awareness and provide training.

According to Lluís Alsedà, who serves as the Director of the Centre de Recerca Matemàtica (CRM) and holds the position of professor at the Universitat Autònoma de Barcelona (UAB), he takes a stance against the JCR index with the following three assertions:

“For years, we’ve been assessing things wrongly. Since the introduction of the JCR index[1], evaluations of both research articles and personnel have been conducted using erroneous criteria. Nevertheless, it’s noteworthy that the institutions of the Spanish State have endorsed the San Francisco Declaration. This demonstrates the willingness to conduct evaluations from a more ethical standpoint, acknowledging the diverse groups involved in research and the varying intensities of research activities. The problem we have now is that we don’t know the rules of the game […] In response to this issue, the Consortium of Academic and Research Assessment (CoARA) has emerged as an entity dedicated to assisting researchers in expressing their opinions. Their goal is to establish a foundation for more ethical and sustainable evaluation methods within the realm of daily research.”

Through active involvement in this united and cooperative endeavour, the Centre de Recerca Matemàtica wholeheartedly dedicates itself to enhancing the assessment of research. The objective is evident: to integrate the instruments established within this alliance, ultimately attaining superior research outcomes of noteworthy influence, all the while upholding equity and openness.

Through this commitment, the CRM will play a part in building a more efficient, inclusive, and advantageous research environment for everyone, paving the way for more meaningful assessments that align with the dynamic needs of the scientific community.

The 10 commitments of the CoARA Agreement

1. 1. Recognize the diversity of contributions to, and careers in, research in accordance with the needs and nature of the research.
   2. Base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators.
   3. Abandon inappropriate uses in research assessment of journal and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.
   4. Avoid the use of rankings from research organizations in research assessment.
   5. Commit resources to reform research assessment as needed to achieve the organizational changes committed to.
   6. Review and develop research assessment criteria, tools, and processes.
   7. Raise awareness of research assessment reform and provide transparent communication, guidance, and training on assessment criteria and processes as well as their use.
   8. Exchange practices and experiences to enable mutual learning within and beyond the Coalition.
   9. Communicate progress made on adherence to the principles and implementation of the Commitments.
   10. Evaluate practices, criteria, and tools based on solid evidence and state-of-the-art in research on research and make data openly available for evidence gathering and research.

For further information about the Coalition, see the CoARA website.

[1] JCR (Journal Citation Report) is a relative quality index (ICR) metric used to assess the scientific quality of academic journals provided by Thomson Reuters. It is determined by dividing the number of citations a journal received in a given year for the two preceding years by the total number of articles published by the journal during those two cited years.