What the eff is TF-IDF, and can it really help your SEO strategy?

You’d be forgiven for thinking, “Those crazy SEO people… what will they think of next?”

But this one isn’t a case of this thought leader or trying to coin a new phrase.

In this chapter, you’ll learn what TF-IDF is, how it works, why it’s part of the SEO lexicon, and most importantly – whether Google uses it as a ranking factor.

The Claim: TF-IDF Is A Ranking Factor

If you go looking to learn more about this topic, you’re going to see some wild headlines designed to make you feel like you missed out by not allocating budget to TF-IDF this year:

• TF-IDF for SEO: What Works & What Doesn’t Work.
• TF-IDF: The best content optimization tool SEOs aren’t using.
• TF IDF SEO: How to Crush Your Competitors With TF-IDF.

Is TF-IDF the SEO tactic you’ve been missing?

The Evidence For TF-IDF As A Ranking Factor

Let’s start with this: what is TF-IDF?

Term frequency–inverse document frequency is a term from the field of information retrieval.

It’s a figure that expresses the statistical importance of any given word to the document collection as a whole.

In plain language, the more often a word appears in a document collection, the more important it is, and the heavier that term is weighted.

What’s that have to do with search?

Well, Google is one giant informational retrieval system.

Say you have a collection of 500 documents and you want to rank them in order of relevance to the term [rocking and rolling].

The first part of the equation, term frequency (TF), is going to:

• Ignore documents that don’t contain all three words.
• Count the number of times each term appears in each remaining document.
• Factor in the length of the document.

What the system ends up with is a TF figure for each document.

But that figure alone can be problematic.

Depending on the term, you could still end up with a pile of documents and no real clues as to which is most relevant to your query.

The next step, inverse document frequency (IDF), gives your TF a little more context.

Document frequency = counting terms across the document collection.

Inverse = Inverting the importance of most frequently appearing terms.

Here, the system removes the term [and] from the equation because we can see that it occurs so frequently across all 500 documents as to be irrelevant to this specific query.

We don’t want documents with the most instances of [and] being ranked highest.

Documents highest weighted for [rocking] and [rolling] while normalizing for text length are more likely to be relevant to people looking for information on [rocking and rolling].

The Evidence Against TF-IDF As A Ranking Factor

As the document collection grows in size and variety, the utility of this metric shrinks.

Google’s John Mueller has spoken about this and explained that

“this is a fairly old metric and things have evolved quite a bit over the years. There are lots of other metrics, as well.”

I don’t think this says it’s not a factor; I think he’s pretty plainly saying it’s just not that important anymore.

And as much as people like to believe Mueller is trying to pull one over on them, there’s no way he’s fibbing on this one.

Identifying which documents contain the words a searcher is querying is a necessary first step in returning a response.

But with that said, it’s an old metric that just isn’t useful on its own.

In an index the size of Google’s, the best that TF-IDF could do is bring back millions or billions of results.

Can you optimize for it?

No.

Trying to optimize for TF-IDF means trying to achieve a certain keyword density, and that’s called keyword stuffing.

Don’t do that.

Still, that doesn’t mean this concept doesn’t matter to SEO pros.

TF-IDF As A Ranking Factor: Our Verdict

Does Google use TF-IDF in its search ranking algorithm – even potentially as a foundational part of its algorithm?

We’re saying definitely not.

Why? Because it’s an ancient (in technological years) information retrieval concept.

Today, Google has far superior ways to evaluate webpages (e.g., word vectors, cosine similarity, and other natural language processing methods).

Knowing whether the word a user is searching for appears in a document and how often is only a first step.

TF-IDF just doesn’t account for much without myriad other layers of analysis to determine things, like expertise, authoritativeness, and trust, for starters.

That means TF-IDF isn’t a tool or tactic you can use to optimize your site.

You can’t do any useful sort of analysis with TF-IDF, or use it to improve your SEO, because it requires the entire corpus of search results to run the calculation against.

Additionally, we’ve graduated beyond simply wanting to know what keywords are used to how they’re used and what related topics come up, to ensure the context and intent matches our own.

SEO pros who use the terms TF-IDF and semantic search interchangeably are misunderstanding TF-IDF.

It’s just a measure of how often a word appears in a collection of documents.

Bottom line: It’s important to understand how content is being evaluated, but that knowledge doesn’t always have to result in another item on your SEO checklist.

Unless you’re building an information retrieval system of your own, TF-IDF is one you can chalk up as an interesting factoid of days gone by and move on.

Featured Image: Robin Biong/Search Engine Journal