Using the real MIND dataset

Why this chapter matters: the capstone shipped with a small, generated sample so it runs anywhere. To build something real (or a portfolio piece), you want a real dataset. This chapter shows how to plug in MIND, the standard real-world news-recommendation dataset, with zero code changes, because we coded to its format from the start.

What MIND is

MIND (Microsoft News Dataset) is a large public benchmark for news recommendation: real news articles and real anonymized user click logs collected from Microsoft News. It comes in two sizes:

  • MINDsmall: ~50,000 users, a manageable few-hundred-MB download. Great for a laptop / this capstone.
  • MINDlarge: ~1,000,000 users, the full benchmark used in research papers.

It naturally includes sports (and soccer/football) alongside many other categories, exactly the themed mix our sample imitates.

Getting it

  1. Download from https://msnews.github.io/ (MINDsmall is the easy starting point). You'll get news.tsv and behaviors.tsv for train and dev splits.
  2. Drop news.tsv and behaviors.tsv into the capstone's data/ directory (replacing the sample).
  3. Re-run the pipeline: 
    python -m newsreco.train          # train + evaluate + log to MLflow
uvicorn newsreco.api:app --port 8000

That's it. No code changes, the loader already speaks MIND.

Why no code changes are needed

Our data loader was written against the exact MIND schema, including its real-world quirks. Verified on a real-format fixture:

articles: 3 (note one has an EMPTY abstract)
empty-abstract ok: ''
interactions (clicks): [('U13740', 'N55528', 1573463158.0), ('U99', 'N61837', ...)]
user U99 empty-history ok: [('N61837', ...)]
times parsed (epoch>0): True

The loader correctly handles:

  • Empty abstracts: some MIND articles have a title but no abstract.
  • MIND's timestamp format: 11/11/2019 9:05:58 AM (note the non-padded hour), parsed to a sortable epoch for time decay and time-based splits.
  • Empty history: brand-new users have a blank history column (a cold-start case the loader and recommender handle).
  • The entity columns: MIND includes title_entities / abstract_entities (named entities); we read the first five columns and ignore the rest, but those entities are a great feature to add later.

What changes at MIND scale (and how the design absorbs it)

MINDsmall runs as-is. As you scale toward MINDlarge, two components need their production swap, both already designed for it:

Concern at scaleSwap (no logic change)
~100k+ articles → exact cosine scan too slowVectorIndex → hnswlib/FAISS (same interface, see ann.py)
TF-IDF vocabulary explodesNEWSRECO_EMBEDDER=sbert → fixed-size neural embeddings
Dense user-item math too bigthe recommender already uses per-item embeddings + ANN, not a dense matrix
k-means / ranker training slowtrain on a sample of impressions (standard practice)

Everything else, the decayed profile, the two-stage ranker, MLflow tracking, the API, the UI, RAG, is unchanged.

A realistic workflow

This is exactly how you'd work in practice, and a good habit to internalize:

  1. Develop on the sample (or MINDsmall), fast iteration, cheap experiments.
  2. Track every run in MLflow (next chapter) so you can compare embedders, half-lives, and ranker settings.
  3. Scale to the full dataset once the pipeline is right, swapping in the ANN backend and neural embeddings.
  4. Validate online with an A/B test before trusting the offline numbers (best practices).

Next: managing trained models with the MLflow registry. 👉