Closing the evaluation gap? Social Sciences and Humanities and the reform of research assessment

Is social sciences and humanities scholarship well served by the current and nascent academic reward mechanisms? And if not, where our options and power lie to change this for the better? Recognized as crucial for the future well-being and flourishing of humanistic scholarship, such questions have been repeatedly raised and discussed on DARIAH Open both in opinion pieces, in declaring commitments or policy reflections. In support of the ongoing European research assessment reform, we are extending this ongoing thread of discussion and inviting expert perspectives from the social sciences and humanities fields to further explore the issues and opportunities our research domain faces in the brave new world of research assessment. We hope in this way to spark a discussion in the arts and humanities on what we can hope to gain, and what opportunities we have to move forward.

In this post, Ioana Galleron, professor of Digital humanities in the University Paris III – Sorbonne Nouvelle and former chair of ENRESSH shares insights on why scholars’ informed judgement from specific disciplinary fields should be in the heart of the ongoing reform, where are the biggest mismatches in the SSH domain in this respect and how the Agreement on Reforming Research Assessment should and could integrate such bottom-up perspectives.

The recent « Agreement on Reforming Research Assessment «, published on the 20^th of July 2022, has been already introduced to the readers of DARIAH blog and underlined as an important step towards realigning research assessment to research realities. Through articulating principles such as the recognition of diversity, whether in terms of outputs, careers or types of contributions, and the abandonment of inappropriate uses of metrics, the signatories of the agreement acknowledge that time has come to alleviate the pressure put on researchers by principles such as “publish or perish”, “publish in JIF journals”, “get international”, etc.

The question remains, though, to what extent the principles stated in the agreement work towards better matching scholars’ vision about what is research quality with what is actually evaluated at the individual, organizational or project level. As previous research has shown, there is a huge difference between what researchers, especially in the SSH, consider as important dimensions of research quality, and what is actually taken into consideration, and sometimes measured, during evaluation campaigns. To give but an example, Ochsner, Hug and Daniel (2012) find that 50% of the criteria articulated by SSH scholars as defining research quality cannot be matched by formal indicators; with another approach, Ochsner and Dokmanovic (2017) demonstrate that criteria put forward by certain evaluation exercises are those least valued by the researchers in the social sciences disciplines.

Through calling to base research assessment on qualitative evaluation, the signatories of the above-mentioned agreement seem conscious of this evaluation gap, and willing to reduce it. Informed judgement about research contributions is, indeed, more apt to infuse discipline-based and epistemically shaped values in the evaluation process, while offering a solution for assessing what cannot be measured. But on should remember that evaluation by peers has its own limitations and pitfalls, that seem somehow forgotten, or overlooked, in the document.

I can easily imagine the discouragement of one (or several) members of the core drafting team, or even of representatives from the signatory institutions, in reading the above statement, that may sound as if I judge the cure somewhat worse than the disease. This is certainly not the case, and I cannot but express my satisfaction for this initiative and its outcomes, as an individual researcher, as a member of the SSH community, and, last but not least, of the European Network for Research Evaluation in the SSH (ENRESSH). However, I cannot help but thinking there is a missing principle in this agreement, that I’d formulate (to respect the rhetoric of the document) as “organize research evaluation from the bottom”. This means starting by understanding how scholars from a specific field conceptualize research quality, and, if necessary, organizing the discussion forum in which, and funding the research activities through which, this conceptualization can be rendered explicit and regularly revised, or even renegotiated.

Indeed, as the agreement goes by, several research quality dimensions are stated: good research advances original ideas, is conducted in rigorous ways, produces results that go beyond the state of the art, is transparent and open, which implies early sharing of data and creating the conditions for verifiability and reproducibility, and is collaborative (including societally engaged). Still, this quite comprehensive list is very limited compared to the number of dimensions of quality stated by scholars during interviews and focus groups where they are invited to express themselves on the topic. It is vital to take into account the diversity of elements participating to the definition of research quality, as well as their dependence to the epistemic frameworks and the evaluative processes (with very different purposes) in which they are mobilized. While it is fundamental to make sure that peers have expertise in the specific research area under scrutiny, that they are transparent and impartial, respect confidentiality clauses and so on, what is even more important is that peers actually have an articulated enough vision about what is research quality, susceptible to be declined in explicit criteria and linked to tangible evidence (whether measurable or not).

While some work has been conducted on the matter, for instance in setting criteria for judging the quality of electronic editions, the DH community has not been, so far, one of the most proactive in reflecting on its own standards. This is certainly linked to the relative youth of a domain in which we all learn, at least to some point, by doing, via a try and error method, and whose objects of study have constantly evolved. Still, it may also be one of the reasons behind the limited recognition DH research is given by certain scholars, stakeholders and institutions. As long as DH scholars themselves cannot oppose structured procedures for evaluating DH research to inappropriate or even disincentivizing evaluation processes, these are here to stay.

The « Agreement on Reforming Research Assessment” is an important step towards improving evaluation procedures so as to render them more efficient in supporting better research. However, its success depends also to its capacity in mobilizing the research communities in defining standards and criteria. As researchers, we should be able to participate in shaping the change. At the very least, we should pay attention to the actual fulfillment of this engagement by those who have taken it, so as to avoid it remaining just lip service.

Ioana Galleron

Sorbonne Nouvelle

Bibliography

Ochsner, M., Hug, S. E., & Daniel, H.-D. (2012). “Indicators for Research Quality in the Humanities: Opportunities and Limitations”, Bibliometrie – Praxis und Forschung, 1, Article 4.

Ochsner, M., & Dokmanović, M. (2017). “Quality criteria and research obstacles in the SSH in Macedonia”, In Book of abstracts of the Second international conference on research

evaluation in the social sciences and humanities, University of Antwerp, p. 69-71

Cite this article as: Ioana Galleron, "Closing the evaluation gap? Social Sciences and Humanities and the reform of research assessment," in DARIAH Open, 25/10/2022, https://dariahopen.hypotheses.org/?p=1630.

---

---