Tech & AI

The Largest Open Datasets for Training AI Language Models

The biggest text collections anyone can download to train an AI language model, ranked by the number of tokens (the word pieces models read) each publisher reports, as of October 18, 2026. Together AI's RedPajama-V2 still leads at 30.4 trillion tokens, with Europe's HPLT 3.0 close behind at about 30 trillion.

Video thumbnail for Together AI's deep dive into the RedPajama datasets

Pictured: video thumbnail for “A Deep Dive Into The RedPajama Datasets” from Together AI's channel; RedPajama-V2 is #1 with 30.4 trillion tokens.

Rank Dataset Tokens Publisher Country First released License Languages
1RedPajama-V230.4TTogether AIUnited StatesOct 2023Common Crawl terms of use5 (English, German, French, Spanish, Italian)
2HPLT 3.0~30THPLT consortiumEuropean UnionJul 2025CC0 (packaging)198 language-script pairs, over half English
3Essential-Web v1.024TEssential AIUnited StatesJun 2025ODC-ByEnglish
4FineWeb18.5THugging FaceUnited StatesApr 2024ODC-ByEnglish
5Dolma 3 (pool)9.31TAi2United StatesNov 2025ODC-ByEnglish, plus code and math
6Nemotron-CC v2 + v2.19.1TNvidiaUnited StatesAug 2025NVIDIA Data AgreementMostly English
7mC46.3TGoogleUnited StatesOct 2020ODC-By101
7CulturaX6.3TUniversity of Oregon & Adobe ResearchUnited StatesSep 2023mC4 and OSCAR terms167
9TxT3605.08TLLM360 (MBZUAI)UAEOct 2024ODC-ByEnglish
10Zyda-25.07TZyphraUnited StatesOct 2024ODC-ByEnglish
11Matrix4.69TM-A-PChinaMay 2024Apache 2.0English + Chinese
12DCLM-Baseline4TDataComp-LMUnited StatesJun 2024CC BY 4.0English
13MADLAD-4003TGoogleUnited StatesSep 2023ODC-By419
13FinePDFs3THugging FaceUnited StatesSep 2025ODC-By1,733 language-script pairs
15Common Corpus2.27TPleiasFranceNov 2024Open licenses, per documentMultilingual, mostly English and French
16The Stack v2 (training set)~900BBigCode (Hugging Face & ServiceNow)United StatesFeb 2024Each repository's own license600+ programming languages
17RefinedWeb (public extract)600BTechnology Innovation InstituteUAEJun 2023ODC-ByEnglish

Basis and caveats: openly downloadable text datasets built for pretraining language models, ranked by the token count each publisher states, as of October 18, 2026. Publishers count with different tokenizers (GPT-2's, GPT-NeoX's, Gemma 3's and their own), which split text into pieces of different sizes, so the figures are approximate and close calls are within the margin. Counts are unique tokens in the released files, not a recommended training mix that repeats some sources (TxT360's “15T+” recipe counts as its 5.08 trillion). One row per dataset family: where a newer version replaces an older one, only the newest is listed (Dolma 3 rather than the 3-trillion-token Dolma of 2023, Nemotron-CC v2 rather than the 6.3-trillion-token v1, HPLT 3.0 rather than v2), and Nemotron-CC v2.1 is counted with v2 because Nvidia publishes it as an add-on to v2. Datasets made by filtering or re-scoring a single other training dataset are left out: FineWeb-Edu, IBM's GneissWeb and the English part of Japan's LLM-jp Corpus v4.1 all come from FineWeb, Dolma 3's roughly 6-trillion-token training mix comes from the Dolma 3 pool, and SlimPajama comes from the first RedPajama. Remixes that merge and re-deduplicate several sources (Zyda-2, CulturaX) count as their own datasets. DCLM-Pool, 240 trillion tokens of raw text pulled from Common Crawl, is treated as raw material rather than a training set, so its filtered DCLM-Baseline is listed instead. Datasets that report words or bytes but no token total aren't ranked: Hugging Face's FineWeb2 (over 3 trillion words in more than 1,000 languages) and EleutherAI's Common Pile v0.1 (8 TB of openly licensed text). “Permissive” means ODC-By, CC0, CC BY or Apache 2.0 terms on the dataset itself; the web pages inside still belong to their authors, and most web datasets also ask users to follow Common Crawl's terms. “Source terms” means the dataset passes on its sources' terms instead. “First released” is when the dataset first came out under that name; sizes are for the current version.[11][12]

The Top 10

#1

RedPajama-V2 (Together AI) — 30.4 trillion tokens

Together AI released RedPajama-V2 in October 2023, and nearly three years later it is still the biggest open training set by stated token count. It holds over 100 billion web documents from 84 Common Crawl snapshots, processed with Meta's CCNet pipeline. After deduplication, its higher-quality “head” and “middle” portions come to about 20 billion documents and an estimated 30.4 trillion tokens: 20.5 trillion in English, 3.0 trillion in German, 2.8 trillion in Spanish, 2.7 trillion in French and 1.5 trillion in Italian. It is designed as a pool rather than a ready-made recipe: 30 billion of its documents carry dozens of precomputed quality signals, so each team can filter it to its own taste. The text itself falls under Common Crawl's terms of use. It followed RedPajama-1T, Together's April 2023 attempt to recreate the 1.2-trillion-token training data of Meta's first LLaMA.[1]

#2

HPLT 3.0 (HPLT consortium) — about 30 trillion

HPLT, short for High Performance Language Technologies, is a research project funded by the EU's Horizon Europe programme and coordinated by Charles University in Prague, with partners including the universities of Edinburgh, Helsinki, Oslo and Turku. Its third release, completed in July 2025, was distilled from about 7.2 petabytes of raw web crawls collected by the Internet Archive and Common Crawl between 2012 and 2024, and covers 198 language-script combinations. The team's paper puts the total at 30 trillion tokens and calls it likely the largest generally available multilingual collection of its kind. More than half is English; the other languages alone come to about 13.5 trillion tokens. HPLT releases its packaging of the text under CC0 and hosts the downloads on its own servers rather than on Hugging Face. It is about three times the size of HPLT's previous release.[2]

#3

Essential-Web v1.0 (Essential AI) — 24 trillion

US startup Essential AI published Essential-Web v1.0 in June 2025: 23.6 billion English web documents from 101 Common Crawl snapshots, globally deduplicated. Its selling point is labelling rather than raw size. Every document is tagged by subject using the Free Decimal Correspondence, an open taxonomy modelled on the Dewey Decimal library system, plus page type, complexity and quality scores. A small 0.5-billion-parameter model did the tagging, which took about 90,000 GPU-hours on AMD MI300X chips. The idea is that teams can carve out a specialist dataset with a simple filter; in Essential AI's tests, a STEM subset made that way scored 24.5% above leading web-curated datasets, and a code subset 14.3% above. It comes under the ODC-By license, alongside Common Crawl's terms.[3]

#4

FineWeb (Hugging Face) — 18.5 trillion

Hugging Face released FineWeb in April 2024 as 15 trillion tokens of cleaned, deduplicated English web text from 96 Common Crawl snapshots (2013 to April 2024), counted with the GPT-2 tokenizer. It began as an open replication of RefinedWeb, the dataset behind TII's Falcon models (#17 here, in its public form), and models trained on it went on to beat those trained on other popular open web sets in Hugging Face's tests. Updates have added newer crawls, most recently six snapshots from January to June 2025 in July 2025, taking it past 18.5 trillion tokens. Its best-known offshoot, FineWeb-Edu, keeps only pages an AI classifier judged educational: 1.3 trillion tokens, or 5.4 trillion with a looser cut. Both are under the ODC-By license.[4]

#5

Dolma 3 (Ai2) — 9.31 trillion

The Allen Institute for AI (Ai2) in Seattle released Dolma 3 with its Olmo 3 models on November 20, 2025. The full pool holds about 9.31 trillion tokens in roughly 10 billion documents: 8.14 trillion from Common Crawl web pages, 972 billion from scientific PDFs converted with Ai2's own olmOCR tool, 137 billion of code, 34 billion of math web pages, 21 billion from arXiv papers and 3.7 billion from Wikipedia and Wikibooks. Olmo 3 itself was trained on a roughly 6-trillion-token mix drawn from that pool, which Ai2 also published. Because Ai2 releases the data alongside the models, Olmo is one of the few model families whose training text anyone can inspect. The first Dolma, in August 2023, had 3 trillion tokens.[5]

#6

Nemotron-CC v2 + v2.1 (Nvidia) — 9.1 trillion

Nvidia published Nemotron-CC v2 on August 18, 2025, at about 6.59 trillion tokens. The largest parts are 3.36 trillion tokens of English Common Crawl text and 1.26 trillion of web pages rewritten by an AI model, plus AI-generated question-and-answer text (some of it translated into 15 languages), math and code. In December 2025 it added v2.1, about 2.5 trillion tokens of new data meant to be used alongside v2, most of it synthetic rephrasing generated with Alibaba's Qwen3-30B-A3B. Nvidia's Nemotron 3 Ultra model card lists the pair at 9.1 trillion tokens. Access requires accepting the NVIDIA Data Agreement for Model Training, which allows commercial use, and Nvidia warns that models trained on it may also fall under the licenses of the models that generated the synthetic text. The original Nemotron-CC had 6.3 trillion tokens, 1.9 trillion of them synthetic.[6]

#7

mC4 (Google) — 6.3 trillion

The oldest dataset in this ranking, mC4 was built by Google to train mT5, a multilingual model described in a paper first posted in October 2020. It applies the filters of Google's English C4 dataset to Common Crawl text in 101 languages, and the mT5 paper counts about 6.6 billion pages and 6.3 trillion tokens. The language detector actually found 107 “languages”, but six were romanized versions of languages already in the set. Ai2 hosts a ready-processed copy on Hugging Face under the ODC-By license. mC4 is also one of the two building blocks of CulturaX, which ties with it at #7.[7]

#7

CulturaX (University of Oregon & Adobe Research) — 6.3 trillion

Researchers at the University of Oregon and Adobe Research released CulturaX in September 2023: 27 TB of text, 6.3 trillion tokens, in 167 languages. It merges mC4 (version 3.1.0) with every release of the OSCAR web corpus from 2019 to 2023, then cleans and deduplicates the result. Access on Hugging Face requires accepting its terms, and it passes on the terms of mC4 and OSCAR. It was built for multilingual models, which have far less data to learn from in most languages than in English, and it has drawn scrutiny as well as use: a 2026 PyData conference talk was titled “Cleaning Large Text Corpora, or Why CulturaX Is Uncultured.”[8]

#9

TxT360 (LLM360) — 5.08 trillion

LLM360 is an open-source AI project started by Petuum and Abu Dhabi's Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), and now run by the university's Institute of Foundation Models. TxT360, short for “Trillion eXtracted Text”, came out in October 2024 as the first dataset to deduplicate 99 Common Crawl snapshots together with 14 curated sources such as research papers, Wikipedia, court opinions from FreeLaw, US patents, European Parliament proceedings and Project Gutenberg books. Its source table adds up to about 5.08 trillion unique tokens, 4.83 trillion of them from the web. LLM360 bills it as a 15-trillion-token dataset because its recommended recipe repeats the best sources several times; this ranking counts each token once. It is under the ODC-By license.[9]

#10

Zyda-2 (Zyphra) — 5.07 trillion

Zyphra, a US AI startup, released Zyda-2 in October 2024, five times the size of its first Zyda. Rather than starting from raw web pages, it combines four existing open datasets, DCLM, FineWeb-Edu, Zyda-1 and the Common Crawl part of Dolma, then removes duplicates across them and filters them with a quality-scoring model. The result is 5.07 trillion tokens by the GPT-NeoX tokenizer, about two-thirds of it from DCLM. Zyphra did the processing with Nvidia's NeMo Curator software, which it said cut the job from three weeks to two days, and trained its Zamba2 models on an early version. It is under the ODC-By license.[10]

In the spotlight USA Google
AI Open Source Datasets Language Models

Sources

  1. Together AI, RedPajama-Data-V2 dataset card (document and token counts, quality signals, license) — huggingface.co/datasets/togethercomputer/RedPajama-Data-V2; Together AI, “RedPajama-Data-v2: An open dataset with 30 trillion tokens for training large language models” — together.ai
  2. HPLT, “HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT” (arXiv 2511.01066) — arxiv.org; HPLT 3.0 dataset card (release date, non-English token count, CC0 terms) — huggingface.co/datasets/HPLT/HPLT3.0; European Commission CORDIS, HPLT project 101070350 — cordis.europa.eu
  3. Essential AI, Essential-Web v1.0 dataset card — huggingface.co/datasets/EssentialAI/essential-web-v1.0; “Essential-Web v1.0: 24T tokens of organized web data” (arXiv 2506.14111) — arxiv.org
  4. Hugging Face, FineWeb dataset card and changelog — huggingface.co/datasets/HuggingFaceFW/fineweb; FineWeb-Edu dataset card — huggingface.co/datasets/HuggingFaceFW/fineweb-edu; “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale” (arXiv 2406.17557) — arxiv.org
  5. Ai2, Dolma 3 pool dataset card — huggingface.co/datasets/allenai/dolma3_pool; Dolma 3 Mix (6T) dataset card — huggingface.co/datasets/allenai/dolma3_mix-6T; Ai2, “Olmo 3: Charting a Path Through the Model Flow to Lead Open-Source AI” (Business Wire, Nov 20, 2025) — businesswire.com; Ai2, “Dolma: 3 trillion token open corpus” — allenai.org
  6. Nvidia, Nemotron-CC-v2 dataset card — huggingface.co/datasets/nvidia/Nemotron-CC-v2; Nemotron-CC-v2.1 dataset card — huggingface.co/datasets/nvidia/Nemotron-CC-v2.1; NVIDIA Nemotron 3 Ultra model card (training data table) — huggingface.co/nvidia; Nvidia Research, Nemotron-CC — research.nvidia.com
  7. Xue et al. (Google), “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer” (arXiv 2010.11934) — arxiv.org; Ai2, C4 and mC4 dataset card — huggingface.co/datasets/allenai/c4
  8. Nguyen et al., “CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages” (arXiv 2309.09400) — arxiv.org; CulturaX dataset page — huggingface.co/datasets/uonlp/CulturaX
  9. LLM360, TxT360 dataset card (source table, ~5T deduplicated tokens, 15T+ recipe) — huggingface.co/datasets/LLM360/TxT360; Middle East AI News, “LLM360 project empowers pre-trainers with huge production ready dataset” — middleeastainews.com
  10. Zyphra, Zyda-2 dataset card — huggingface.co/datasets/Zyphra/Zyda-2; Zyphra, “Building Zyda-2” — zyphra.com; Nvidia, “Train Highly Accurate LLMs with the Zyda-2 Open 5T-Token Dataset Processed with NVIDIA NeMo Curator” — developer.nvidia.com
  11. Ranks 11–17, from each publisher's dataset card or paper: M-A-P Matrix (huggingface.co/datasets/m-a-p/Matrix); DCLM-Baseline and DCLM-Pool (huggingface.co/datasets/mlfoundations, arXiv 2406.11794); Google MADLAD-400 (arXiv 2309.04662, huggingface.co/datasets/allenai/MADLAD-400); Hugging Face FinePDFs (huggingface.co/datasets/HuggingFaceFW/finepdfs); Pleias Common Corpus (huggingface.co/datasets/PleIAs/common_corpus); BigCode The Stack v2 (huggingface.co/datasets/bigcode/the-stack-v2, arXiv 2402.19173); TII RefinedWeb (arXiv 2306.01116, huggingface.co/datasets/tiiuae/falcon-refinedweb)
  12. Datasets left out of the ranking: Hugging Face FineWeb2 (word counts only) — huggingface.co/datasets/HuggingFaceFW/fineweb-2; IBM GneissWeb — huggingface.co/datasets/ibm-granite/GneissWeb; LLM-jp Corpus v4.1 — gitlab.llm-jp.nii.ac.jp; EleutherAI, “The Common Pile v0.1” — eleuther.ai