Bringing arts and humanities perspectives to the redefinition of ” what counts” in research(er) evaluation

Research assessment (i.e. decisions on allocation of research funds, academic career advancement, and the hiring of staff) has been recognized as the Achilles heel of firmly grounding Open Science practices in research realities for a long while now. In academia scholars are still facing conflicting injunctions and have to walk on both paved and unpaved paths while advocacy and research policy efforts repeatedly point out enormous complexities, systemic impediments and failed attempts in upscaling thoughtful, alternative proxies that could replace the the current harmful system dominated by publisher prestige. 

As an example of the many voices urging and supporting this change, in DARIAH’s response to the the stakeholder consultation on the  Future of scholarly publishing and scholarly communication  European Commission report in 2019, we argue that the vicious circle  in which research evaluation is lingering can only be broken through a set of urgent and harmonized actions and call for a new social contract on the European level involving funders, research performing institutions and their ministries, university networks, disciplinary communities and research infrastructure providers (including publishers).

The vicious circle of research evaluation. Source: https://doi.org/10.5281/zenodo.4922538.

Realigning research assessment with digital and diverse research realities: a big step closer in 2022? 

Entering 2022, we see strong European-level policy drive and momentum  to indeed finally )put such a new social contract in place for reforming research assessment based on EU-wide consensus and to turn the long preparatory work into a set of actual, tangible  implementation cases  that are flexible enough for upscaling them or show interoperability with an emerging European common ground.  Most notably – and in alignment with other ongoing EU efforts such as the Research Careers, Recognition, and Credit EOSC task force or Horizon Europe calls for putting the infrastructural dimensions of the reform in place -, the European Commission’s Scoping Report Towards a Reform of the Research Assessment System (Nov. 2021), declaring practical commitments from the Commission and serving as a compass for downstream implementation routes. 

The document projects forward an ambitious vision of  meaningfully accommodating a lot more and badly needed diversification in evaluation proxies in terms of showing sensitivity to national and regional flavors, institutional stakeholder perspectives, disciplinary and epistemic cultures, career stages,  and a variety of research missions while still designing a system, or set of systems, that are efficient and flexible in the sense of showing interoperability across all these dimensions, to support mobility of researchers or multidisciplinary research. All this suggests that in order to turn  this long-awaited research assessment reform into reality, the different actor groups involved are facing an awful lot of work to do – starting immediately. 

4 ways in which DARIAH supports the ongoing European reform of research assessment

At DARIAH, we recognize this balance between diversity and efficiency as a crucial success criteria. In this post, we put forward priority areas and practical commitments and showcase how we play our part in this important cause, in areas that naturally fall close to DARIAH’s mission. 

Tóth-Czifra, Erzsébet, & Papaki, Eliza. (2022). Bringing arts and humanities perspectives to the redefinition of ” what counts” in research(er) evaluation. https://doi.org/10.5281/zenodo.5902333 .

1. Informing European-level policy decisions by evidence-based advocacy; ensuring that arts and humanities  scholars are well represented in the emerging systems 

Good quality research is done in diverse places and in diverse ways. Making sure that the new generation research assessment frameworks and metrics are well aligned to domain-specific epistemic traditions is an absolute priority for DARIAH. This usually entails translation of the generic principles and values to the open research culture to disciplinary realities and finding equivalents between cornerstone issues in STEM and arts and humanities disciplines. For instance, both the dominance of Impact Factor in STEM fields and the defining career weight on where one publishes their first monograph in arts and humanities are harmful instances of the same phenomenon of publisher prestige still serving as a primary proxy for research assessment. 

Source: Emaniel Kulczycki: How to prepare early career researchers
for the two sides of research evaluation? https://www.eua.eu/downloads/content/emanuel_kulczycki.pdf

To identify good practice and clear evidence of what works and what doesn’t and provide policy guidance to support innovations based on them, DARIAH have been continuously investing in meta-research and assessments within our communities. In addition to the DARIAH Impact Factors and Success Criteria Working Group and insights gained from collaborations with a diverse range of research teams in training events and externally funded research projects, in 2019-2020, DARIAH-EU led  a task force on research assessment and peer review innovations in SSH within the H2020 project OPERAS-P. The results of the task force will be published as an Open Access monograph. The gained insights became directly translated in service development (see point 3) as well as policy bodies and consultations. For instance, in 2019, Dr. Jennifer Edmond led the task force on next-generation research indicators  within. the Open Science Policy Platform and since 2021, Erzsébet Tóth-Czifra is contributing to the Career, rewards, recognition EOSC task force. All these efforts are carried out in alignment with SSH innovators in research assessment, such as HUMetriccs, ENRESSH  and sister ERICS. 

Due to DARIAH’s organization structure with currently 20 ministries among our investors and a significant number of research performing institutions as member organizations, DARIAH can provide real scenarios of piloting and implementing next-generation research scenarios as they make sense in the arts and humanities domain. 

Resources

Task force on next-generation research metrics at the Open Science Policy Platform

Research assessment and peer review innovations in SSH within the H2020 project OPERAS-P 

DARIAH Impact Factors and Success Criteria Working Group Career, rewards, recognition EOSC task force

2. Strengthening evaluation communities around valuable scholarship that is yet invisible in formal assessment

Recognizing and celebrating the diversity of research outputs, their publication fora as well as their audiences, still finding ways to flexibly accommodate them into interoperable and functional research assessment frameworks is a key challenge tackled by DARIAH. Throughout the whole Ats and Humanities domain, but especially in Digital Humanities, a wide range of outputs that are important for scholarly communities are still invisible from the formal peer review or research evaluation horizont (such as digital scholarly editions, research software and research environments built by scholars, exhibitions, digital analysis tools, databases, archival guides etc.). To change this for the better, DARIAH is dedicated to solidify evaluation frameworks around certain types of digital scholarship and build bridges between evaluation culture around them and formal research assessment. For instance, in 2018, DARIAH launched the OpenMethods platform to facilitate critical discourse on Digital Humanities tools and Methods, or in 2021

Resources
OpenMethods platform 

Evaluation framework for the assessment of lexicographical datasets in collaboration of ELEXIS 

Evaluation framework for the assessment of Virtual Research Environments in Digital Humanities 

Annual Open Access book bursary for first monographs in Digital Humanities

3. Supporting scholars at all career stages in working using open knowledge practices – and in recognizing their value and contributions to high-quality research 

Although, as pointed out below,  the infrastructural dimensions required for the successful and systematic reform of the current research evaluation proxies are by far non-negligible, changing harmful proxies for ones that are better aligned to diverse, digital and open research practices is a cultural change in the first place. To facilitate this change in culture across arts and humanities scholars and research support professionals, DARIAH regularly provide training events, workshops and other co-creation opportunities to help scholars open a bigger window on their work or evaluate new types of scholarship, or conduct and assess research in terms of the integrity of processes, rather than purely in terms of product/outputs. Relevant training materials are captured, indexed or hosted on the DARIAH Campus training platform. Another line of advocacy in this respect is to connect scholars with innovative publication fora such as networked monographs, data journals or overlay journals so that they can integrate a broader range of their work with information management systems used for formal research assessment. 

Resources

Regular online, offline training in topics like evaluation of data and code, data citation, research ethics, peer review, data and overlay journals etc. 

Capturing, indexing, hosting these training materials on the DARIAH Campus platform

4. Contributing to shared infrastructures for gathering of high quality open data for research assessment and expanding their inclusiveness with arts and humanities content 

Research evaluation cannot live without shared scholarly information management systems that allow scholars, institutions, and funders to keep track of the increasing amount of scholarly production year by year. By now, the shortcomings and in some cases, even intransparency of the first generation of such services (Scopus, Web of Science etc.) as well as their enormous impact on assessment practices became obvious and gave rise to an ever pressing need to replace or at least complement them by publicly owned information management systems that are based on transparent processes and open data traffic and are inclusive with a broad range of content types and disciplines. 

DARIAH has made two major investments in this direction so far. First, through our participation in the OpenAIRE project, we connected thematic data services from the national levels to the OpenAIRE Research Graph that serves as European data commons and as such, is expected to play an important role in research administration too. This way, content types that are important for the arts and humanities communities (digital critical editions, audiovisual data, encoded documents, images etc.) became visible in the biggest publicly owned scholarly discovery information management system. 

Second, in collaboration with sister ERICS such as CLARIN and CESSDA, DARIAH led the development of the Social Sciences and Humanities Open Marketplace (henceforth SSHOC Marketplace), an SSH-specific discovery system dedicated specifically to research tools, services, data, and workflows – again,  content types that are yet invisible in formal discovery and assessment mechanisms and systems. The marketplace, released in December 2021, is also expected to strengthen the evaluation culture around these entities. However, the fact that contributions to this flagship infrastructure component, operated by and for researchers are currently not rewarded by institutional and other formal evaluation proxies pose a crucial challenge to the long-term well-being and sustainability of such services (these are for instance data curation, evaluating entities or ingesting new workflows to the system). 

Resources

Adding arts and humanities components to OpenAIRE Explore which is  probably the biggest such publicly owned, scholarly discovery service in Europe

Leading the development of the Social Sciences and Humanities Open Marketplace. Both endeavors bring services, tools and content types to the radar of shared European information management systems that were previously invisible for formal assessment but that are important for scholarly communities in the arts and humanities. 

Innovation still disincentivized? Stakes are high – not only for the scholars we are serving but also for DARIAH themselves 

As it is encoded in DARIAH’s mission statement (“to empower arts and humanities research communities with digital methods to create, connect and share knowledge about culture and society”), a central aim of our organization is to provide support structures: opportunities, networks and infrastructure for scholars to explore, build, test and sustain together digital solutions to upscale their research methods and change (expand?) the ways in which culture can be accessed and analyzed. Such activities take different forms of co-creation, such as participation in DARIAH-sustained working groups, becoming editors of or curating content in DARIAH services such as OpenMetods or the SSHOC marketplace, teaming up to build databases or research software, participating in training events both as trainers or trainees, evaluating submissions for the DARIAH Annual Event or funded calls, editing publications,  experimenting with novel technologies etc. 

Source: https://doi.org/10.5281/zenodo.5498024

As the above example of the SSHOC marketplace  at point 4 already puts forward, only a small fraction of these scholarly activities are visible on the horizons of formal research assessment, the defining majority of them is still unrewarded, if not even counter-incentivized by the  traditional bibliometric and publisher prestige-based mechanisms of assessment. 

In other words, DARIAH’s mission of bringing on collaborative and innovative means of digitally-enabled research cannot be fully realized unless the conflict between the richness of contemporary scholarship and the prestige economy that defines our current academic evaluation culture is not mitigated. With this in mind, DARIAH remains truly dedicated and invested in representing arts and humanities communities in projects and initiatives to innovate research assessment. 

Cite this article as: Erzsébet Tóth-Czifra, "Bringing arts and humanities perspectives to the redefinition of ” what counts” in research(er) evaluation," in DARIAH Open, 18/01/2022, https://dariahopen.hypotheses.org/1172.


You may also like...

1 Response

  1. 07/04/2022

    […] kolei w tekście „Bringing arts and humanities perspectives to the redefinition of >what counts< in research(er)…” (2022) Erzsébet Tóth-Czifra opisuje działania infrastruktury DARIAH-EU, które wspierają […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search OpenEdition Search

You will be redirected to OpenEdition Search