Workflows for Creating, Managing and Archiving Textual Corpora in Under-resourced Languages

The DARIAH Working Groups Research Data Management and Multilingual DH joined their forces this year in a research project financed by the DARIAH Working Groups Funding Call 2023-25 entitled “Creating, Managing and Archiving Textual Corpora in Under-resourced Languages”. In the context of this project, the Working Groups recently organised a workshop which took place both in-person in Hamburg University (28-30 August 2024) and online. The purpose of the workshop was to develop one or more workflow(s) to be published on the Social Sciences & Humanities Open Marketplace – SSHOC.

The participants were also invited to submit their workflows in Transformations DARIAH Journal , the first issue of which is dedicated to workflows.

The quality of the presentations was really impressive and it became evident that both working groups are sharing overlapping concerns and facing common challenges in creating a managed corpora, for example tools and methods for corpora digitisation, sustainability, legal aspects (sensitive information), versioning, corpora and metadata structure,  documentation, visibility and long-term archiving, risk to loose specificity of the language.

Throughout the three-day event, participants of the workshop from different disciplines and countries produced collectively and progressively three workflows and submitted them in the SSHOC Marketplace. The three following workflows: “Creating, Managing and Archiving” are interlinked, interacting with each other as in a loop.

1. Building/Creating Textual Corpora in under-resourced languages

This workflow explains the different steps you should follow in order to create your textual corpora, from searching a text collection preferably in Europeana to legal and ethical aspects of your corpus.

Within this workflow, you should anticipate the question of digitisation (tools for OCR) and romanisation of your data (if non-latin-script), as well as file formats you want to work with (e.g. XML/TEI, Json). All your actions should be described in a Data Management Plan (DMP). This workflow is to be used with the “Managing Textual Corpora” and “Archiving Textual Corpora”.

https://marketplace.sshopencloud.eu/workflow/67KJnp

2. Managing Textual Corpora in under-resourced languages

This workflow is to be used with the “Creating Corpus” and “Archiving Corpora” to have a complete set of passages to management of corpora for under-resourced languages.  The “Corpus Management” workflow provides a set of steps to manage your corpora while staying compliant with FAIR and Open Science principles. The emphasis of this workflow will be on the documentation of all corpora dimensions and the creation of metadata and use of controlled vocabularies. Best practices, dedicated tools and references to other workflows are provided.

https://marketplace.sshopencloud.eu/workflow/fbXfuH

3. Archiving Textual Corpora in under-resourced languages

This workflow presumes that you already created a textual corpus together with a data management plan and supporting documentation, and that you are ready to deposit it in a data repository.

This workflow explains how to deposit your corpus data, metadata and documentation either in a CLARIN-B data centre, or in your university’s institutional research data repository, or in a free-for-everybody repository like Zenodo.

This workflow is to be used with the “Managing Textual Corpora” and “Building Textual Corpora”.

https://marketplace.sshopencloud.eu/workflow/gDzIoY

Cross-cutting issues and Open Science

During the workshop, participants expressed ideas and opinions on different issues starting with the question of the target audiences of the workflows. All participants converged that these workflows are mainly addressed to multilingual researchers and linguists.

From an Open Science perspective, all workflows presented during the session stressed the very central position of documentation of the corpora and structure of metadata which subsequently enable findability and interoperability of corpora. At the same time, some participants had quite bad experience in terms of sustainability of the platform used to archived their corpora which led us to envision backup strategies (as data migration) to avoid data loss. Such workshops where workflows are shared and formalised is crucial to allow researchers and institutions improve their collaboration, in particular in the domain of under-resourced languages so as to reduce the invisibility of languages beyond English in Digital Humanities.

Contributions, contact and references

• wg-rdm[at]dariah.eu
• multilingualdh[at]dariah.eu

All contributions and contents of the workshop are available in GitLab: DARIAH Working Groups Research Data Management and Multilingual DH, Creating Managing and Archiving Textual Corpora in Under-resourced Languages (Hamburg, 2024), https://gitlab-ce.rrz.uni-hamburg.de/uahh-digitale-dienste/creating-managing-and-archiving-textual-corpora/ . The grant has been administered by Universität Hamburg. Lead applicant was Francesco Gelati (he/him). Co-applicants were Alíz Horváth and Maroussia Bednarkiewicz.