Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Assessing non-traditional academic formats: the hidden traps to the treasure trove

people searching information

People searching for some information

From the result to the process

The advent of digital methods has contributed to raising awareness of hitherto neglected methodological bricks and activities adjacent to (and relevant for) the production of knowledge. Academia traditionally values research outputs published in an authoritative version like monographs and research papers. Yet, digital approaches have shown that iterative processes as well as the intermediate work steps are necessary to write the research papers in the first place. Formalising methodologies as command lines for computational processing purposes means that the underlying methods and tools need to be made explicit and reproducible.

This shift towards explainability represents a change of paradigm: the focus is not solely on the final research output, the context and process to get there are just as relevant. It is now generally acknowledged that data modelling, for instance, is a major aspect of the research process. This is all the more important as all research relies, in one way or another, on data work – even in the Humanities.

Although there is a general awareness of the multiplicity and variety of the methodological tooling that is indispensable to conceive and complete research endeavours, the bulk of academic credit remains reserved to traditional formats. Prior history in research assessment makes it possible to rely on what is usually considered as a solid validation procedure. Humanities scholars tend to consider that with a monograph, they know where they’re at. The situation is fairly different when it comes to less traditional research outputs. Here, the scholarly communities lack guidance: Which bricks in the research process deserve to be valued? How can their value be assessed? How can it be made more visible, and benefit those involved in their development?

Assessing non traditional research outputs

Recently, several European initiatives have engaged in the elaboration of a framework of reference for the evaluation of non-traditional research outputs in order to advance the recognition of forms of academic labour that have long remained below the surface in the vast ocean that is academic recognition. Open Science has been instrumental in this progress by fostering transparency and reproducibility, but it is crucial now that the research communities embrace these principles beyond theory and transform them into actionable requirements within their community of interest.

Digital-based Humanities research is disseminated in a plethora of formats – some well established, others still in their infancy. Criteria for the evaluation of digital scholarly editions and publication venues for such evaluations, have been around for several years now, and they have contributed to set standards at a point in time where such editions were bound to become a low-quality by-product of work on historical documents. This example shows that it is key to identify the areas where defining evaluation criteria and developing publication formats and venues is likely to impact the concerned fields of research towards higher quality standards. 

 

ATRIUM logo

In the context of the ATRIUM project, we are working on the one hand on establishing a publishing model that makes it technically possible to publish non-traditional academic outputs – Transformations. A DARIAH Journal intends to be truly transformative in that regard. On the other hand, we are also working on a peer-review framework to provide rigorous criteria for the evaluation of these outputs. The ultimate goal is to provide academic recognition in a manner that mirrors the actual involvement of the roles and skills involved in producing research. 

 

Evaluation criteria for non-traditional formats

The first sets of evaluation criteria concern themselves with data papers, workflow papers and training materials. While we were striving to develop a common assessment framework that would be somehow aligned for all of these outputs, we came to the conclusion that these were too different and ultimately required different sets of assessment criteria.

Data papers are a fairly established format, practiced by a variety of journals. A data paper consists of a paper that describes and contextualises a dataset. While these datasets used to be provided in a closed format for reviewing purposes in natural sciences journals that first developed this type of publications, Open Science encouraged a shift towards transparency so that datasets were made openly available and reusable whenever possible. The reviewer of a data paper is not expected to review all the data or underlying code, but they are to consider a sample of it as well as their condition of availability additionally to the assessment of the paper that is describing them. 

Workflow papers are similar to data papers in the sense that they provide a description and contextualisation of information that is provided in another format. Things are a bit trickier here in the sense that workflows, as opposed to data, are not a clearly established format. Developing an evaluation framework for papers that present workflows requires reflecting on what workflows are in the first place. Technically, a workflow is an actionable research pipeline – to be made accessible to users, it needs to be made explicit through a description of the steps, data and tools involved. A workflow paper is hence multilayered as it accounts for the relevance and efficiency of the workflow, the clarity of its description for reuse purposes, and the quality of its contextualisation. The reviewer is expected to consider all these elements and the way they are technically connected: here, the task is really novel, and we envision workshop formats to familiarise both potential authors and potential reviewers with challenges and benefits of this type of publication.

Training materials involve other challenges. Since they are from the onset expected to specify their context of use, they don’t formally require an additional layer of text explaining this context like it is the case with data papers or workflow papers. This poses in turn questions of the visibility of the academic recognition: should the training materials themselves be included in a journal? If so, in which form? While the overlay format chosen for the journal Transformations facilitates the integration of data papers and workflow papers (as the publication process is backed up by open repositories), it raises conceptual issues when it comes to training materials – issues that are bound to be relevant for other types of research outputs as well. 

 

Idealism in practice

Providing a framework for the recognition of the invisible data labour in the Humanities is a noble goal, and it certainly requires some idealism to engage in its pursuit – but it also requires to address issues of technical feasibility, of priorities to be set, of the publication infrastructures that can back the intellectual work in a coherent and sustainable manner. It needs all actors in the academic system to be rowing together – a true challenge on the way to unlocking the potential awaiting to be discovered.


OpenEdition suggests that you cite this post as follows:
Anne Baillot (June 18, 2025). Assessing non-traditional academic formats: the hidden traps to the treasure trove. DARIAH Open. Retrieved January 23, 2026 from https://doi.org/10.58079/145f7


You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.