Introducing SimpleStories

To interpret deep neural networks, one needs to answer two questions simultaneously: “what to look for and how?”. It is both unknown what deep model’s internal mechanisms are and how they are represented. This stems for a lack of structure in most datasets, containing random scraps of data from the internet. Hence, there is increasing need for structured datasets that can help guide interpretability research toward concrete goals.

One milestone toward this is TinyStories, a dataset containing millions of children’s stories. Tiny language models (~10 million parameters) trained on this dataset can generate coherent and creative stories. The combination of tiny models and constrained problem space forms a perfect testbed for understanding these models.

Unfortunately, this dataset suffers from a few issues:

• Stories are generated using old models, leading to formulaic and/or incoherent plots.
• Some stories are ‘corrupted’, containing large amounts of nonsense characters.
• The vocabulary isn’t constrained enough, leading to complex or even misspelled words.

Furthermore, there are ample opportunities for improvement:

• Fine-grained labels about aspects of the story (tone, moral, topic) that can serve for supervised probing/finetuning.
• Strong focus on simplicity, while retaining diversity.

Keeping this in mind, we created SimpleStories.

@misc{finke2025parameterizedsynthetictextgeneration,
      title={Parameterized Synthetic Text Generation with SimpleStories}, 
      author={Lennart Finke and Chandan Sreedhara and Thomas Dooms and Mat Allen and Emerald Zhang and Juan Diego Rodriguez and Noa Nabeshima and Thomas Marshall and Dan Braun},
      year={2025},
      eprint={2504.09184},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.09184}, 
}