{
  "video_id": "reddit_1uswlq8",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Training an LLM from scratch on 1800's texts (160GB dataset)",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1uswlq8/training_an_llm_from_scratch_on_1800s_texts_160gb/",
  "external_url": null,
  "upload_date": "20260710",
  "published_at": "2026-07-10T18:51:37+00:00",
  "transcript": "Hi everyone, A year ago I began pre-training language models exclusively on 1800’s London data. Recently I have completed my largest dataset ever, containing 40B tokens or 160GB of 1800-1875 english data from England and the United States. I will soon train a 2B parameter model on it, but for now I’ve trained a 500M parameter evaluation model on a 5B token sample. I have also fine tuned the eval model on 1800’s Q&A pairs (using synthetic questions and answers pulled straight from the dataset), so you can ask it about historical figures, places, events, etc. It works better with  London stuff for now and it’s not that accurate since it’s just an eval model but the results are promising for a larger run. \n\nSome sample outputs: \n\nhttps://preview.redd.it/ncw7d62g6gch1.png?width=1020&format=png&auto=webp&s=2fbdb24b8db90c1c36a7ba4a30238da46ed802f6\n\nThe recipe generation for the plum pudding is insane, so hopefully the 2B model won’t tell you to stir with your feet.\n\n[https://github.com/haykgrigo3/TimeCapsuleLLM](https://github.com/haykgrigo3/TimeCapsuleLLM)\n\n[https://huggingface.co/haykgrigorian/TimeCapsuleLLM-English-1800-1875-v3mini-eval1-500M](https://huggingface.co/haykgrigorian/TimeCapsuleLLM-English-1800-1875-v3mini-eval1-500M)\n\n\n\n--- Top Comments ---\n\n\n[45 upvotes] This looks amazing!! Waiting for the 2B model\n\n[32 upvotes] Update the bias report in v3 to racial bias, the world wants to know....  \n  \nthis thing could be more racist than Grok \n\n[19 upvotes] this is the kind of project that makes r/LocalLLaMA great. have you tried asking it about events that happened AFTER 1875 to see how it handles questions outside its training window? like does it confidently make up a plausible 1800s-sounding answer about the telephone (patented 1876) or does it just not know? a model with a hard knowledge cutoff at 1875 is basically a built-in hallucination test. also the 2B model trained on the full 40B tokens is going to be wild... 500M on 5B tokens is already generating plum pudding recipes, can't imagine what 8x the data does to the quality.\n\n",
  "transcript_chars": 2070,
  "ingested_at": "2026-07-11T01:30:09.872141+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 190,
    "upvote_ratio": 0.98,
    "num_comments": 46,
    "author": "Remarkable-Trick-177",
    "is_self": true
  }
}