A recommendation system predicts which items a user is most likely to want and presents them in a product experience that makes acting on them easy. This article explains the three modelling approaches, the data problems that cap performance, and how to evaluate a live system. It is written for product and data teams planning or fixing a recommendation capability.
Key takeaways
- A recommendation system has two halves, the engine that ranks items and the product experience that presents them, and both decide its commercial value.
- Collaborative filtering needs dense interaction history, content-based filtering needs consistent item attributes, and production systems are usually hybrid to cover the gaps.
- Data quality caps recommendation quality, so fixing catalogue attributes and planning for cold start often lifts results more than changing the model.
- Offline metrics such as precision and recall shortlist candidates, while controlled experiments on conversion, engagement and catalogue coverage decide what ships.
The question we are asked at the start of these projects is almost always about the algorithm. Which model should we use, and is the one our competitor uses better. It is the wrong first question in a way that costs months.
In the systems we have built and inherited, the recommendation engine has not been the reason performance was disappointing. The reasons were sparse interaction data, a catalogue with inconsistent attributes, or a placement in the interface that nobody looked at.
What is a recommendation system?
A recommendation system is a machine learning system that ranks items for a specific user based on what that user and similar users have done before. It has two halves: the engine that produces the ranking, and the product experience that decides where the ranking appears, how many items show, and what happens when a user ignores them.
The term recommendation engine usually refers to the first half alone. Distinguishing the two is useful, because they fail for different reasons and are owned by different people. An excellent engine placed below the fold produces nothing. A mediocre engine placed well and refreshed often can carry real commercial value.
The three modelling approaches
Three approaches cover the large majority of production systems, and the right one depends on the data you have rather than on which is most sophisticated.
Collaborative filtering predicts what a user will like from the behaviour of users who resemble them. It needs a substantial history of interactions and performs badly on new users and new items.
Content-based filtering predicts from the attributes of items a user has already engaged with. It handles new users better and depends entirely on the quality of your item metadata.
Hybrid approaches combine the two, typically using content-based signals to cover the gaps where collaborative data is thin. Machine learning recommender systems in production are usually hybrid, because pure approaches break at the edges where real catalogues live.
| Approach | Data required | New user behaviour | New item behaviour | Best fit |
|---|---|---|---|---|
| Collaborative filtering | Large interaction history | Poor, no signal to work from | Poor, no interactions yet | Mature catalogues with high traffic |
| Content-based | Rich, consistent item attributes | Good, works from first interaction | Good, attributes exist immediately | Catalogues that change often |
| Hybrid | Both, in usable quality | Good | Good | Almost all production systems |
Google’s recommendation systems course covers the mathematics behind each approach for teams that want the detail.
Why the data decides the outcome
Recommendation quality is bounded by the quality of the interaction data and the item metadata underneath it, and no model choice raises that ceiling.
Four data problems account for the failures we see:
Cold start. New users and new items have no history. Every system needs an explicit answer for what to show them, and defaulting to bestsellers is a decision worth making deliberately.
Sparse interactions. In a large catalogue with modest traffic, the interaction matrix is mostly empty. Collaborative approaches need more density than many organisations actually have.
Inconsistent item attributes. Missing categories, uneven titles and absent filter attributes limit content-based approaches directly. Fixing catalogue data is unglamorous and frequently produces a bigger lift than changing the model.
Feedback loops. A system that recommends what it previously recommended successfully narrows over time. Deliberate exploration is what keeps the catalogue visible. The GroupLens research group has published on this problem for longer than almost anyone.
Matching in practice
The same principles apply to matching systems, which rank people against opportunities rather than items against users.
We built one with EHL, Europe’s leading hospitality education provider, as part of a talent matching and learning platform. The platform connects individuals to short-term roles across hospitality while offering training modules developed and certified by EHL, giving businesses a way to fast-track recruitment and upskill existing teams at the same time.
Our team designed an intelligent matching system combining a talent marketplace with lifelong learning modules, built around both hard and soft skills. Definition and design began in June 2021, the first version released in June 2022, and the service offering extended through further releases across the second half of that year.
What that project reinforced is that ranking quality depends on how well the underlying attributes are structured. Matching people to roles requires an agreed vocabulary for skills before any ranking is meaningful, in the same way that recommending products requires a consistent catalogue.
How to evaluate one
Evaluate on business metrics, and use offline metrics only to decide what is worth testing live.
Offline measures such as precision and recall tell you how well a model reproduces historical behaviour. That is useful for filtering candidates and misleading as a final answer, because reproducing history is not the goal. Run the shortlist as a controlled experiment and measure the metric you actually care about: conversion, engagement depth, or repeat visits.
Watch the guardrails alongside the headline metric. A system that lifts click-through while narrowing the range of items customers ever see is storing up a catalogue problem, and that shows up months later as declining discovery rather than as a failed test.
One further measure is worth tracking from the start: coverage, meaning the proportion of your catalogue the system ever recommends to anyone. Low coverage tells you the model has settled into a small popular core, which is comfortable in the short term and expensive for any business whose margin sits in the long tail.
Key takeaways
- A recommendation system has two halves, the engine that ranks items and the product experience that presents the ranking, and both determine commercial outcome.
- Collaborative filtering needs dense interaction history, content-based filtering needs consistent item attributes, and production systems are usually hybrid.
- Recommendation quality is bounded by data quality, so improving catalogue attributes often produces a larger lift than changing the model.
- Cold start, sparse interactions, inconsistent attributes and narrowing feedback loops account for the failures we see most often.
- Offline metrics filter candidates for testing, and business metrics measured in a controlled experiment decide what ships.
Planning a recommendation capability?
We help teams work out whether their data supports the system they want, fix the foundations where it does not, and build what sits on top. Talk to our data and AI team about what you are trying to rank.
__________________________________________________________
The Groundwork Advantage
Between 60 and 95 per cent of enterprise AI initiatives produce no measurable business value. Our free AI handbook covers what has to be true inside an organisation before AI produces anything: clear business objectives, data you can trust, governance built in from the start, and infrastructure that keeps your options open.
Read the handbook | Take the five-question self-assessment
__________________________________________________________
