Behind the Scenes of Transformations: A Conversation Between Author, Reviewer and Editor on the Making of Issue 1

From the submission process until final publication, authors have to face and respond to many requirements of the academic publishing process and even more for a digital humanities journal open to non-traditional research outputs. From writing the text itself to ensuring that the outputs are reusable, authors are invited to follow rigorous steps and recommendations throughout the editorial process (see our Author Guidelines) to ensure that their final contribution is high-quality, original and innovative. Assessing the scientific quality of the submissions is not an easy task for the evaluator either and it is one of the core missions of the publisher to establish accurate and transparent processes. This is one of the tasks of the Editorial Board, who has to implement and provide the reviewers with a solid methodology to assess a diversity of research outputs. This includes clear criteria on the contextualisation of the research question, the state of art, the coherence and clarity of the text, and the reusability of the results.

Gathering experiences from the publishing process of the first issue of Transformations on the topic of Workflows, this blog post aims to present some of the challenges that emerged in this process from the perspectives of the author, the reviewer and the Managing Editor. These three roles are very much representative of the continuous and enriched dialogue between these actors throughout the publication process.

In this interview, we welcome one published author: Adeline Clarke (University of Helsinki), one reviewer: Bianca Gualandi (University of Bologna), one managing editor: Anne Baillot (member of Transformations’ Editorial Board). The questions are addressed alternately to the author, the reviewer or the Managing Editor, with all providing either a direct answer or feedback. The reviewer selected for this conversation did not assess the author’s paper, and the nature of the feedback is more generic. We hope this dialogue will reflect the diversity and richness of the exchanges between these stakeholders throughout the editorial process from submission to publication.

All submission types in Transformations are assessed in accordance with a shared editorial quality framework encompassing four core sets of criteria: state of the art, content, form, and accessibility and reusability. In this interview, we focused on the cross-cutting challenges that affect both authors, reviewers and editors: 1) contextualisation of the research question; 2) accessibility and reusability of data and tools; 3) clarity and linguistic accessibility; and 4) metadata structure and quality. 

Contextualisation of the research question

(See Section 4 on Research Question, Author Guidelines)

Question to the author (Adeline Clarke, University of Helsinki)

How did you manage to situate your contribution within a broader disciplinary context?  What methodology did you implement in your paper and can you explain how you justified its originality? 

Author’s response (Adeline Clarke):

Our paper introduced our tool finnsurveytext which is an R-package for use in analysing open-ended survey questions. While initially intended for use with Finnish-language survey responses, the tool works for many languages. The R package was initiated as a work package within DARIAH-FI, an infrastructure project for the social sciences and humanities in Finland, and so Transformations was deemed an appropriate venue for a paper about the package.

The tool was created to meet a need of social science and humanities researchers in Finland for a free and open-source tool for analysing text from survey data in Finnish which did not require advanced programming tools. As such, the tool was situated within survey analysis as a complement for more-established methods for analysing closed questions. We justified the originality of the tool by situating it in contrast to the limited other tools available for this purpose which were largely commercial tools with limited functionality.

Reviewer feedback (Bianca G.):

Both articles I reviewed situated their contribution within the relevant disciplinary context, although with different degrees of detail. In the introductory sections, they reviewed already established practices in the respective fields, highlighting the existing gaps and presenting the main challenges they were seeking to address. Although different in scope, in my opinion they both effectively addressed the existing landscape and convincingly justified the originality of their approach.

Editor feedback (Anne B.):

The finnsurveytext article revolves around a tool that was developed for a different context than the arts and humanities scholars that are the primary audience of the journal. The authors provided a very precise contextualisation, showing the benefit, not only of the tool itself but of the challenges connected to its development, for arts and humanities scholarship. To me, it is one of the major strengths of this paper. Incidentally, it is also an excellent example that Transformations is not just “a digital humanities journal”, but a journal that strives to broaden the epistemological horizons of arts and humanities by providing novel perspectives on the contribution of digital methods at large.

Accessibility and reusability of data and tools

(See Section 6 on Data and Tools, Author Guidelines)

Question to the editor (Anne Baillot, DARIAH-EU)

One of Open Science’s key principles is reproducibility. Ensuring that data and tools are reusable means that other researchers can replicate and re-analyse data or apply findings in new contexts. 

Bearing this in mind, how did you ensure that the data and tools published in this volume were reusable? Did you use any particular strategies or tools to verify this? 

Editor’s response (Anne B.):

This is a tricky question. While there are well-established conventions for citing books and articles, there are still no clear or universally accepted standards for citing data and digital resources in general. Yes, we know that it is better to provide a persistent identifier, but in the end there are often format and infrastructure constraints. Actual reproducibility requires an extensive set of metadata and is in a way still subject to interpretation. As editors, we approached this candidly at first but realised in the course of the editorial work that we needed to be extremely careful and precise in what we were requiring from authors. We now have an editorial task in place that is dedicated solely to verifying elements ensuring reproducibility and data availability, and if they are not fulfilled, the paper will not be published. It is one of the points where we will be much stricter from issue 2 onward – the first issue was in many ways a demanding learning process!

Author feedback (Adeline C.):

In the case of my article, it was very easy to ensure and show reproducibility. As part of the submission process to the CRAN (Comprehensive R Archive Network), the R language’s central software repository, it must pass a number of automated checks which ensure reproducibility of example output. Similarly, the example data, which was used in the article to produce the example output, is included within the finnsurveytext package.

Reviewer feedback (Bianca G.):

This is a tricky question from the point of view of reviewers, too! As discussed, they should not only check that data and tools underlying the article they are reviewing are available (es. through a data availability statement), but also try to re-use them and replicate the findings. In my own reviews, I have done my best to be thorough – pointing out if data and tools, or their metadata, were insufficient or missing – without fully replicating the studies as it would have been impossible in the given timeframe. Unfortunately, we all know that reviewers’ time is precious and they are increasingly hard to find!

Can we collectively think of ways to make it easier for reviewers to replicate the results of the papers they are called to assess? And what about the scenarios in which full replicability is not possible? When is an article replicable, or transparent, enough?

Clarity and linguistic accessibility

(See Section 7 on Linguistic Accessibility, Author Guidelines)

Question to the reviewer (Bianca Gualandi, University of Bologna)

Authors in Transformations are often writing about complex workflows and technical tools, which are not traditionally published as journal articles. 

How did they approach explaining these technical aspects clearly and coherently? Do you think they succeeded in adapting their arguments for their intended audience?

Reviewer’s response (Bianca G.):

I reviewed more than one paper for Transformations and – as you would expect – the authors had different levels of clarity and coherence in explaining the more technical aspects of their work. I consistently tried to point out the passages that were not easy to follow and needed additional work.

Open research is not only concerned with writing up results in neat “final” publications but also with showing the inner workings that have allowed the authors to achieve those results. This is why workflow descriptions and technical articles are increasingly important in every discipline and I hope that venues like Transformations become more common.

Author feedback (Adeline C.):

We aimed for our writing to be accessible and clear since the purpose of the article was to introduce and explain the tool to potential users who could span a variety of backgrounds and levels of experience using R packages. To be honest, I often found this difficult and it required many iterations with my co-authors. It is difficult when you are so familiar with the tool to take a step back and try to see it through someone else’s eyes!

I was very grateful for the extensive and generous feedback that the two reviewers of my paper provided which I believe resulted in a more polished, easier to follow, and more persuasive article.

Editor feedback (Anne B.):

There are several dimensions to clarity and accessibility. The first point concerns linguistic competence. As we chose Academic English as a publication language for this issue (a multilingual workflow would require resources that we don’t have at this point, as much as we would like to), as editors, an important part of our work was to make sure that the manner in which the arguments were expressed really did justice to the complexity of the arguments themselves. Second, bearing our audience in mind, we asked for improvements both for specialists and for non-specialists. In other words, there has been quite a lot of editorial work on top of the reviewing itself, which served as a basis for quality check. Some authors were somewhat surprised by these numerous iterations. Looking back at issue 1, we can see that this demanding work is absolutely worthwhile. The papers all hold up to a very high quality standard.

Metadata structure and quality

(See Section 8 on Metadata and Image Rights, Author Guidelines)

Question to the author (Adeline Clarke)

Transformations places strong emphasis on high-quality, well-structured metadata to make articles easy to find and access across databases and repositories. How did you make sure your article included complete and accurate metadata, including image captions and usage rights, from the very start of the submission process?

Did you consider it as an easy step of the editorial process? If not, what challenges or obstacles did you face, and what improvements would you suggest from an author’s point of view?

Author’s response (Adeline C.):

As part of the submission process, we were required to put our paper in an open archive such as HAL. As such, our paper was deposited on Zenodo, which facilitated much of the metadata requirements of the journal. Unfortunately I missed a few requirements here, but the Transformations journal staff were very helpful in pointing out what was missing. The Zenodo platform was very user friendly and it was easy to add this metadata.

In terms of usage rights for tables and figures used within the article, we were lucky as all of the examples within the article came from sample data which is included with the package (and this was sourced from the Finnish Social Science Data Archive and provided under licenses which allowed the data to be shared for such purposes). Similarly, all the images came from package output and thus were allowed to be shared.

Therefore, we found, partly due to the nature of our article and the finnsurveytext tool, that the metadata requirements of the journal were very easy to meet!

Reviewer feedback (Bianca G.):

Working as a data steward, I’m very used to looking for mistakes in metadata! This may be why I found it quite natural, during my review, to also concentrate on the article metadata, as well as those of tools and data that accompanied them.

Metadata are crucial for retrieving, connecting and making sense of scholarly outputs. It is therefore very important that authors are made aware and are trained on how to produce correct, complete and accurate metadata as part of the submission process. Reviewers of course can play their part by devoting attention to metadata, suggesting potential additions and improvements.

Editor feedback (Anne B.):

Realising the importance of the metadata requirements was a big learning of the editorial board of the journal during the reviewing and copy-editing phase of the first issue. In the case of the finnsurvey article, there were no major difficulties, but for others, oftentimes we had to control the quality and iterate several times with the authors – the devil is so often hiding in the details! Obviously, authors wish to concentrate on the actual content of their publication, and even metadata-aware people don’t write articles with the goal of providing perfect metadata, but an article that will be read and cited. Metadata quality is one area that we, the editorial board, have to be particularly meticulous about, but since we are not the ones responsible for the original deposits, ultimately, the responsibility remains in the hands of the authors.

Interviewees: Anne Baillot (DARIAH EU), Bianca Gualandi (University of Bologna), Adeline Clarke (University of Helsinki). Reviewers: Eliza Papaki, Megan Black, Toma Tasovac (DARIAH EU). Coordination: Françoise Gouzi (DARIAH EU, Member of Transformations’ Editorial Board).

 

 

 

 

 

 

 

---

The text only may be used under licence Creative Commons Attribution 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.

---

---