Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research)
Subscriber deck — free during early access
1 cards in “2.2. The Colossal Clean Crawled Corpus” · whole deck
2.2. The Colossal Clean Crawled Corpus
- How was C4 cleaned from Common Crawl?Keep lines ending in terminal punctuation; drop pages with fewer than 5 sentences and lines under 3 words; drop pages with words from a bad-words list, “lorem ipsum”, or curly brackets (code); drop lines mentioning Javascript; deduplicate repeated three-sentence spans; keep pages detected as English with probability ≥ 0.99. The result is about 750 GB.
- Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 2.2. The Colossal Clean Crawled Corpus, pp. 5–7
Source
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. In Journal of Machine Learning Research, pp. 1–67. License: CC BY 4.0. Flashcards adapted from this work. https://jmlr.org/papers/v21/20-074.html