Hugging Face 推出了 FineTranslations，这是一个庞大的多语言平行文本数据集，包含超过 1 万亿个 token，在英语和 500 多种其他语言之间进行了精细对齐。该数据集通过使用 Gemma3 27B 模型将 FineWeb2 语料库中的非英语部分翻译成英语而构建，整个数据生成过程完全可复现且公开记录。FineTranslations 的主要目标是增强机器翻译能力，特别是针对英语到其他语言的翻译方向，这在低资源语言中通常表现较差。通过利用原本为非英语语言的文本并将其翻译成英语，该数据集提供了大量平行数据，非常适合微调现有的翻译模型。除了翻译之外，内部评估显示，翻译后的英语内容保留了源语言的重要文化和上下文信息，为纯英语模型预训练提供了高质量的补充，其表现可与 FineWeb 本身相媲美。
Hugging Face has released FineTranslations, a large-scale multilingual dataset containing more than 1 trillion tokens of parallel text across English and 500+ languages. The dataset was created by translating non-English content from the FineWeb2 corpus into English using Gemma3 27B, with the full data generation pipeline designed to be reproducible and publicly documented.
The dataset is primarily intended to improve machine translation, particularly in the English→X direction, where performance remains weaker for many lower-resource languages. By starting from text originally written in non-English languages and translating it into English, FineTranslations provides large-scale parallel data suitable for fine-tuning existing translation models. Internal evaluations also indicate that the resulting English text performs on a similar level to FineWeb for English-only model training, allowing the data to be reused beyond translation-specific tasks.
Beyond translation, Hugging Face reports that the resulting English corpus retains substantial cultural and contextual information from the source languages. In internal experiments, models trained on the translated English text achieved performance comparable to those trained on the original FineWeb dataset, suggesting that FineTranslations can also serve as a high-quality supplement for English-only model pretraining.
The dataset is sourced from FineWeb2, which aggregates multilingual web content from CommonCrawl snapshots collected between 2013 and 2024. To reduce skew toward highly repetitive or domain-specific material, such as religious texts and Wikipedia pages, only language subsets with a bible_wiki_ratio below 0.5 were included. For each language, up to 50 billion tokens were processed, with quality classifiers from FineWeb2-HQ applied where available, and random sampling used otherwise.
Translation was carried out at scale using the datatrove framework, which enabled robust checkpointing, asynchronous execution, and efficient GPU utilization on the Hugging Face cluster. Documents were split into chunks of up to 512 tokens, with a sliding-window strategy to preserve context across segments. Additional safeguards were introduced to mitigate common large-scale translation issues, including early classification of toxic or spam-like content, strict formatting constraints, and post-processing to ensure consistency of line breaks and structure.
Each dataset entry includes aligned original and translated text chunks, language and script identifiers, token counts, quality and educational scores, and references to the original CommonCrawl source. The dataset can be accessed through the Hugging Face datasets library, streamed for large-scale processing, or consumed directly using datatrove-based pipelines.
Commenting on the release, Achref Karoui said:
Awesome! This release will bridge the gap and allow communities to better align popular models with their languages.
FineTranslations is available now on Hugging Face. The dataset is released under the Open Data Commons Attribution (ODC-By) v1.0 license, and its use is subject to CommonCrawl’s terms.