Talking peer review series #2: Experiences in Optimizing Peer-Reviewing – A success story in five acts, three years, and two astonishing voting rounds

Peer review is a central scholarly practice, a practice that carries an enormous weight in terms of gatekeeping, shaping disciplines, publication patterns and power relations. It governs the (re)distribution of resources such as research grants, promotions, tenure and even larger institutional budgets. As such, it is crucial to deeply understand how it works in situated evaluation practices, and to continually rethink it to strive for its best, and least imperfect (or reasonably imperfect) instances. This new series on the blog, Talking Peer Review, is a means to do just that. You will find here reflections, reports, opinion pieces and, to bring in some metascience, reviews of the current state of the art in the arts and humanities disciplines. In this episode Anne Baillot, Professor in German Studies at Le Mans Université shares her and her team’s experiences on bringing open peer review to the annual conference of the German-speaking Digital Humanities association – in 5 acts. Her insights do not only shed light on some of the dramatic aspects of peer review and showcase why it is difficult to change ingrained, established practices; but also reveal how an empathetic, deeply sensible approach can foster meaningful dialogues across members of a scholarly association and can eventually lead to the redefinition of such practices along increased transparency and equity.

I remember when we first talked about opening the reviewing process for the annual conference of the German-speaking Digital Humanities association (Digital Humanities im deutschsprachigen Raum ). At first, it was part of a larger discussion on the improvement of data management within our conference management system (ConfTool). Then it turned into a topic in and of itself. In the first round of discussion that we held at a member’s meeting, I think it was in 2019, I was disappointed when I saw that many good friends were against giving up on single blind peer review. This was reflected in the first voting round we had in 2020. Especially women, post-docs voiced concerns that open peer reviewing would put them in awkward or weaker positions and increase their unease. So let this be said before I dive into the story I want to tell: you can *always* refuse to do a review. You don’t even have to give a justification: if it makes you feel uncomfortable, it is a conflict of interest, you send it back and somebody else should do it. The only thing you need to take into consideration is the reshuffling effort that sending back reviews requires from the Program Committee, e.g., you should check your conflicts of interest as soon as a review is assigned to you, and not the day before the deadline. 

This being said, let’s move to Act one of our story. The German-speaking Digital Humanities community is certainly not as large as other research communities, but it remains the largest Digital Humanities community worldwide. Still, and certainly due in part to the hierarchical structure of German-speaking Academia, DH developed around geographically well identified centers and big names who contributed to the reputation of places like Hamburg, Cologne, Trier, Leipzig or Darmstadt. The result is that we all know each other – or a close enough neighbour. Plus, we work in Digital Humanities. In other words: we develop online resources, and logically refer to them in our proposals. This means that it requires quite some effort to anonymize proposals in such a way that double blind peer-reviewing would make sense. And so we had been working mostly with single blind peer-reviewing for several years of conferencing, but the idea to engage in an open conversation between author and reviewer was not too far-fetched. It was submitted to discussion and voting at the member’s meeting in 2020, where it roared from all sides.

In the wake of this heated discussion, the DHd board commissioned a group of people to work on guidelines. The group was initially called “Open Peer Review task force” because the point was to be able to make a case for Open Peer Review. Still, and luckily, it was not composed of Open Peer Review supporters only. Its actual strength comes from the fact that it was composed of supporters both of open and of “closed” review processes, who battled for two years within what became the “Optimized Peer Review task force”. 

Act 2, the task force got to work. They organized a workshop and crafted their first publication: a 5-page document with recommendations for the review process of the next conference (DHd2022). Reading the “Handreichung” was then recommended to all reviewers of the conference. The recommendations deal with constructive criticism and conflicts of interest; they also give detailed input on the reviewing criteria and how to implement them. As an experienced reviewer, I particularly appreciated the first part (on constructive criticism) that manages to put the reviewer in the author’s shoes and helps them reassess how a review can be perceived by an author. AndI am convinced that the last part of the recommendations can be of great help for those who are new to the business of reviewing. The “Handreichung “was not the only arrangement submitted by the task force to the board and Program Committee: they also suggested awarding a “Best Review” Prize (which was implemented in DHd 2022), and to have an Ombudsperson for conflict situations (which was not realized in the setting initially envisioned, but was operational). In fact, the task force envisioned the optimization of peer reviewing as a way of bringing the community together, of strengthening the ties. Training young scholars in reviewing practices was also part of the deal.

Act 3, the experimentation begins. For the DHd 2022, among the many flavours of open peer review (as defined by Tony Ross-Hellauer),  we decided to implement the model with open identities i.e., in which author and reviewer know each other. What is more, a 2-step rebuttal phase was integrated (similar to what had been done for DH 2020, without the final “response to the response to the rebuttal” round): in order to foster a scholarly dialogue, authors have the opportunity to write back to the reviewers, and reviewers have the possibility to take these answers into consideration e.g., modify their scores and their reviews. This conversation is still restricted to plain text paragraphs within the reviewing management system, but it gives a chance to contribute additional elements, add links, specify how a reference suggested by the reviewer can be integrated, etc. My personal experience is that, as a reviewer, I actually modified scores and reviews based on the authors’ responses, and as an author, I was somewhat disappointed when after having worked with my co-authors on a lengthy answer to a very critical review, the reviewer didn’t even check our answer. 

The first shot was not perfect, but still: the numbers speak for themselves in terms of improving scholarly communication and quality. According to the task force, the number of reviewers grew significantly (from 122 in 2020 to 199 in 2022 – there was no conference in 2021 due to COVID), which was more than welcome considering the growth in proposals (177 in 2020, 219 in 2022). All in all, 714 reviews were written, with an average of 3,26 reviews per reviewer and 222 words per review (an increase of 33% compared to 2020). The average score awarded to submissions was significantly improved (+ 5%). But what I consider more important is the array of the scholarly conversation that took place: 399 rebuttal comments were submitted (with an average of 168 words per comment), and 67 reviews were modified in the context of the rebuttal phase. 

Act 4, all of you who have studied drama theory know we are not so far from the resolution but there is some delaying event to come first. Believe it or not, the delaying moment is the conference itself. On day one of DHd2022, I attended the workshop given by the task force that gave me a good idea of what training for novel reviewers could look like. The workshop setup was extremely well-thought in terms of pedagogy, and the practical parts brought together experienced and less experienced researchers in tandems. We actually worked on anonymized reviews and try to settle whether longer reviews really are the better reviews. While it turned out that short reviews are definitely not helpful, longer reviews are not necessarily optimal either: some are too general, some too nicely formulated and don’t reflect the score, some address a topic that is not really that of the proposal, some lack structure… Should optimizing peer reviewing not mean learning (and if necessary, re-learning) to write a review in the first place?

Two days later, a panel organized by the task force presented arguments in favour of open identities in the peer review process and arguments against it. It was obvious by then that none of the sides in the task force had changed their minds after two years of meetings and discussions. The panel finally presented the voting proposal that had been submitted to the board: the members were to vote on a choice of 3 reviewing models for next year’s conference: Open identities like 2022, single blind like the years before, or double blind. One open variation and two closed ones, none of them pushing the boundaries of openness further than they had already been taken. I was not very optimistic regarding the outcome of the vote.

Moving on to Act 5, where you learn to trust your community… Yes, we voted in the first round and with a clear majority for Open Identities. Try it, and you’ll adopt it? That is one possible conclusion you can draw from this experience. But I think the trick was not just to impose a model, but to build a discussion, training settings and contradictory debate around it for the community to grow into it. Kudos to the board and to the task force (Manuel Burghardt; Lisa Dieckmann; Svenja Guhr; Nils Reiter; Walter Scholger; Timo Steyer; Peer Trilcke; Ulrike Wuttke) – because there is a major effort behind such a result. Community-building also means helping your community grow into the challenges of its time. 

Thinking of this in terms of conference management systems for a conclusion, there would also be a lot to say. We could do so much in terms of sociology and epistemology of our research practices, of our research communities, if only we could work with reviewing data (e.g., topics, reviews, evolution of grading practices, historical semantics,…). This would require to make the peer reviewing even more optimized: to open the reviews themselves, to tune tools like ConfTool for information extraction (building upon the amazing work that already allows an import from ConfTool to Zenodo like for DHd2022) and anonymization of data subject to GDPR to foster shareable formats for reviewing platforms. But the first step is to accept to open the black box of evaluation. And the German-speaking DH community seems to be moving on with seven-league boots on that road.

---

---