Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Introduces two openly licensed datasets:
1. SwallowCode (≈16.1 billion tokens) refines Python snippets from The-Stack-v2
2. SwallowMath (≈2.3 billion tokens) enhances Finemath-4+ by removing boilerplate, restoring context, and reformatting solutions into concise, step-by-step explanations
abs: https://arxiv.org/abs/2505.02881
datasets: https://huggingface.co/datasets/tokyotech-llm/swallow-code
https://huggingface.co/datasets/tokyotech-llm/swallow-math
