ATRIUM — ÚFAL documentation¶

The Institute of Formal and Applied Linguistics (ÚFAL, Charles University) builds six tools for ATRIUM. Together they take archival documents, such as typewritten reports, handwritten field notes, photographs, drawings and born-digital files, and turn them into structured, searchable records. This site explains what each tool does and how they fit together.

What ATRIUM is¶

ATRIUM (Advancing fronTier Research In the arts and hUManities) is an EU project. It connects four European research infrastructures: DARIAH (arts and humanities), ARIADNE (archaeology), CLARIN (language technologies) and OPERAS (open scholarly communication). More at atrium-research.eu, and in the project presentation on Zenodo.

This site documents ÚFAL's part: six tool repositories and the hub that holds them together, seven repositories in all.

Start here¶

Find the line that describes you:

You are	Start with	Then
archive or collection staff	For archives	Licences and paradata · Glossary
running the services	Operations	S2 pilot handover, which links each service's runbook
calling a tool from your own code	Workflows	The document contract · Schemas · Agent skills
developing the tools	Architecture	Contributing standards · Development history
managing or coordinating the project	Status	S2 pilot handover · Repository map
judging the methods and results	Pipelines	each tool's evaluation pages, from the documentation map in its README

Or by question:

If you want to…	Read
look up an unfamiliar word	Glossary
see what each tool does, step by step	Workflows
see what passes from one tool to the next	Pipelines
know what a record written by these tools looks like	The document contract
run a tool as a service, or on Kubernetes	Operations
let a coding agent use a tool	Agent skills
know what every field and label means	Schemas · SKOS & the ATRIUM vocabulary
publish results to a repository or catalogue	RO-Crate export
describe a tool's workflow on the SSH Open Marketplace or in Galaxy	Workflows
read about a standard, model or service the tools use	External tools & services
find which repository owns what	Repository map
change shared code, or contribute	Architecture · Contributing standards

The seven repositories¶

page-classification

Sorts each scanned page into one of eleven page types. A person or a script then decides whether the page needs OCR, handwriting recognition, table extraction or image handling.

Workflow · Documentation · Landing page · Source
ocr-postprocess

Checks OCR output line by line. Every line gets its language and a quality label, so you know which lines to use, correct or send to OCR again. It reads ALTO and the other common OCR formats, and PDF, office and text files.

Workflow · Documentation · Landing page · Source
translator

Translates ALTO pages and AMČR metadata records in place. Every tag, namespace and coordinate stays; only the text changes.

Workflow · Documentation · Landing page · Source
nlp-enrich

Adds each word's base form, part of speech and grammar, and the names of people, places and organisations, using LINDAT's UDPipe and NameTag. The result is a TEITOK file that keeps every word's place on the page.

Workflow · Documentation · Landing page · Source
keyword-extract

Gives documents two kinds of keyword: statistical ones taken from the text, and terms from AMČR's keyword lists and the TEATER thesaurus, chosen by a language model.

Workflow · Documentation · Landing page · Source
digital-convert

Reads born-digital files (PDF, DOCX, ODT, ODS, XLSX, RTF; DOC and XLS through LibreOffice) into the document record, without OCR. Pages whose text cannot be read are marked, so only those go to OCR.

Workflow · Documentation · Landing page · Source
atrium-project, the hub

The code every tool shares, the checks they all run, the end-to-end tests, and this site.

Architecture · Repository map

How the tools work together¶

flowchart LR
  SCAN[/"scanned pages"/] --> PC[page-classification]
  PC -. "routing decision" .-> OCR["OCR<br/>(outside the pipeline)"]
  OCR --> ALTO[ocr-postprocess]
  DOCS[/"office and text files"/] -. "text-lines" .-> ALTO
  BORN[/"born-digital PDF, DOCX"/] --> DC[digital-convert]
  ALTO -- "PAGE_ALTO/" --> TR[translator]
  ALTO -- "DOC_LINE_CATEG/" --> NLP[nlp-enrich]
  NLP -- "record" --> KW[keyword-extract]
  DC -- "record" --> KW
  TR --> EN[/"English editions"/]

Six tools, each in its own repository. Each tool is a command-line program and an HTTP service built from the same code. Each ships as container images. The tools need very different environments: a vision-model stack, CPU rules, remote translation services, GPU language models. So they are not combined into one program.

One record per document. A JSON file, <doc_id>.document.json, ties the stages together. Each stage reads it, adds the one block it owns, and passes it on. By the end it holds the page categories, the text, the translation reference, the entities and the keywords. It also records which program, at which version and with which components, wrote each part. See The document contract.

One hub. atrium-project holds the code every tool shares: the record, the paradata, the licence catalogue, the vocabulary and the service contract. It hands out identical copies of that code, runs the checks every tool calls, and runs the end-to-end test of all the tools together. See Architecture.

Built here, run by the partners. ÚFAL builds, tests and publishes the container images. The partner institutes of archaeology run them on their own infrastructure, next to their own collections. See Operations.

About this site¶

Every page here is written, not generated. It explains how the parts fit together, and ends with a Sources table naming what it was written from. Each tool's own documentation, its README and its docs/ pages, holds the details; these pages connect them.

The site's source is docs_site/ in the hub. A pull request builds it with mkdocs build --strict, which fails on any broken link or anchor between pages. A merge to main publishes it to the gh-pages branch.