Challenge
High-quality simplification datasets are scarce outside English. The project needed to turn noisy, document-level comparable corpora into traceable sentence pairs suitable for training and evaluating language systems.
Selected work
Multilingual data engineering · 2026
Align and Shine
A production-minded data pipeline for constructing aligned text-simplification corpora across five languages.
My role
I designed and optimised extraction and alignment workflows for large-scale shared compute, keeping provenance, intermediate artefacts and reproducible releases visible.
Outcome
Released an open five-language data product, reduced a core workflow from 48 to 6 hours and supported a peer-reviewed paper presented at BUCC, LREC 2026.
At a glance
5languages aligned
48 → 6hcore workflow runtime
87.5%runtime reduction
System view
The workflow.
A simplified view of the stages and boundaries that shape the project.
- 01Comparable documents
- 02Normalisation
- 03Sentence embeddings
- 04Alignment
- 05Quality checks
- 06Open corpus
Approach
Decisions that shaped the work.
Treat the corpus as a data product
The workflow keeps intermediate artefacts and provenance visible, rather than hiding the process inside a single notebook.
Compare alignment strategies
The repository brings together multilingual encoders and alignment methods, including BGE, LaBSE and SONAR-based workflows.
Engineer for shared compute
Batch execution, compression and SLURM-oriented processing made it possible to run the workflow efficiently on shared Linux and GPU infrastructure.
Findings
What the evidence says.
- Released aligned simplification data for Catalan, English, French, Italian and Spanish.
- Created a consistent basis for cross-lingual text-simplification experiments.
- Documented the work in an open repository with separate code and data licensing.
Publication and release
Align and Shine: Building High-Quality Sentence-Aligned Corpora for Multilingual Text Simplification
Peer-reviewed paper in the official proceedings of the 19th BUCC workshop at LREC 2026. The paper begins on PDF page 72 of the workshop volume.
Engineering reflection
The next engineering step is to add a root-level quickstart, automated tests and CI around a small fixture corpus so contributors can validate the full pipeline without access to the shared compute environment.