Joining forces to realign research assessment to research realities: a Europe-wide Agreement on Reforming Research Assessment is now published 

Research outputs now encompass far more than what can be placed on a bookshelf. Knowledge advancement happens in much more diverse places and are brought about by much more diverse processes and dynamics of co-creation than what has traditionally been recognized.  For this reason, the existing tools of academic rewards and recognition criteria, such as citation counts, h-indexes, and the weight of publisher prestige have ceased to accurately reflect what we most value in, and need from, research.  This misalignment severely affects the well-being and innovation potential of academia, as it prevents the widespread adoption of open research practices, works against research integrity, forces scholars working with digital methods to do the same work twice and in general, hinders the democratization of higher education. 

While the need for a fundamental rethinking of academic incentives has been notoriously voiced in a wide range of disciplinary communities, it has also become clear from previous, failed attempts that a reform of research assessment criteria will only be possible through a synchronic and harmonized action across all parties involved. 

An important milestone: publication of the European Agreement on Reforming Research Assessment in July 2022

On the 20th of July, the European Commission, Science Europe and the European University Association (EUA) reached a milestone in bringing a critical mass of 380 organisations (universities, funders, academies and learned societies, research centres, ministries, etc.) together from across and beyond Europe to form a coalition and carry out this timely reform. The first result of the joint effort, the Agreement on Reforming Research Assessment, sets shared goals and priorities for this critical mass of stakeholders on how to reform assessment practices for researchers, research projects and research performing organisations. 

The envisioned reforms are defined alongside the following core commitments:

1. Recognise the diversity of contributions to, and careers in, research in accordance with the needs and nature of the research
2. Base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators
3. Abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index
4. Avoid the use of rankings of research organisations in research assessment

The full text of the agreement gives further context, a detailed description of the commitments and associated timeframes of the endeavour. An accompanying document containing Frequently Asked Questions (FAQ) on the Agreement, future coalition and next steps was also published.

Timeline and next steps 

The agreement itself is a result of a co-creation process that started in January 2022, under the leadership of a core drafting team  and with the participation of organisations from 40+ countries who expressed their interest in being part of a coalition on reforming research assessment. The agreement was drafted during the first half of the year iteratively, and has been reviewed within the framework of assembly meetings with potential coalition members. 

As a next step, from September 2022 on, collection of signatures to the Agreement will be launched. Signatories are expected to adhere to the key commitments of the agreement within their own, autonomous institutional practices and further shape their implementation by mutual learning and evidence-based feedback/advocacy. 

Making sure that the evaluation needs of arts and humanities disciplines are well addressed

DARIAH has been contributing to the initiative from the beginning by voicing shared needs and epistemic specificities of the arts and humanities domain and emphasising the need for discipline-tuned infrastructural enablers to such a process throughout the drafting process and the assembly meetings. As a natural continuation of this line of work, DARIAH had expressed early interest in signing the agreement. Dr. Jennifer Edmond, President of the Board of Directors of DARIAH said: 

“If Open Science in Europe is to progress, we need to move beyond a focus on the outputs of research to the processes that create it. This is particularly the case for the arts and humanities, where workflows and forms of expression can be distinct and inherent indicators of the quality of research. For this reason, DARIAH has warmly welcomed the initiative to form a multi-stakeholder approach to the reform of research assessment, and we look forward to helping to shape its implementation”  

Incentivising innovation, digital and open research practices and equitable academic governance models is a key driver not only for the disciplinary communities we serve but also for DARIAH itself. A central aim of our organisation is to spark and support novel digital methods and new forms of co-creation through which cultures and societies can be studied. Currently, only a small fraction of these scholarly activities are visible on the horizons of formal research assessment, the defining majority of them is still unrewarded, if not even counter-incentivised by the  traditional bibliometric and publisher prestige-based mechanisms of assessment.

This post gives an overview of priority areas and practical commitments of our organisation and showcases how we play our part in this important cause, in areas that naturally fall close to DARIAH’s mission.

As we welcome this new initiative and its implications over the coming months, we will be exploring further the issues social sciences and humanities face in the brave new world of research assessment. We hope in this way to spark a discussion in the arts and humanities on what we can hope to gain, and what opportunities we have to move forward in this new world. 

Links

• The Agreement
• Additional Information on the Drafting Process and List of Organizations involved
• 20220720 RRA Agreement Coalition FAQ

---

---