Hopefully, you’ve heard about IndexNow at this point.
We covered its announcement in October and in December, I had Fabrice Canel, Principal Program Manager for Microsoft Bing, on the SEJ Show to talk about this revolution in how search engines discover new and updated content.
But what is it, exactly, and what do you need to know about it for SEO?
Here’s what we know so far.
Why We Need IndexNow
Search engines have been discovering content the same way for a long time.
Way back in 1993, JumpStation became the first internet discovery tool to send out bots to follow (or crawl) links from website to website and create a content map, or index.
Within the next few years, other search engines adopted this process as a way to find new content.
This process was included in an algorithm that also considered content, relevancy, and other factors – and search engine optimization as we know it was born.
It didn’t take long before SEO-savvy web publishers recognized the importance of indexing in securing high search engine rankings. In fact, without indexed content, a website won’t generate organic traffic.
But therein lies the rub: Unless you’re a large, highly trafficked website, it may be days or even weeks before a search engine crawls your site and indexes new pages. Heck, even if you are a large ecommerce site or marketplace, it may take engines weeks to pick up new arrivals or out of stock products as well – especially when depending on discoverability with basic crawls.
Meanwhile, you may have lost important traffic to competitors.
To streamline this process, Microsoft Bing released a new initiative in October 2021 called IndexNow.
So What Is IndexNow?
IndexNow is an open-source protocol that allows website publishers to instantly index across participating search engines, updating results based upon the latest content changes.
Simply put, it’s a simple ping that notifies search engines that a URL and its content have been added, updated, or deleted.
By eliminating the need for exploratory crawls, IndexNow seeks to improve overall internet efficiency.
How Does IndexNow Work?
There are two different ways search engines obtain index data: Pull and push.
Pull indexing occurs when a search engine visits your site to request web pages and gathers data from the server.
This is how search engines traditionally operate.
Push indexing is when the web publisher or content management system notifies the search engine that pages have been added or deleted or other changes have been made.
That’s what’s truly groundbreaking about the IndexNow – it allows all URLs submitted to any IndexNow-enabled search engine to be simultaneously submitted to all the other search engines using the protocol.
You can read more about the technical aspects of how IndexNow works here.
Which Search Engines Use IndexNow?
At the time of publication, Bing and Yandex are using IndexNow.
A handful of others are reportedly testing the protocol, according to an unnamed expert.
So far, DuckDuckGo has not announced any plans to adopt IndexNow.
However, it’s worth noting that the privacy-focused search engine draws from up to 400 sources in addition to using its own DuckDuckBot web crawler in the compilation of its search results.
Given that Bing and Yandex are among those sources, you could see updated content appearing in DuckDuckGo quicker due to IndexNow, as well.
Even Bing’s largest competitor, Google, the foremost of all search engines, have confirmed they will be testing the IndexNow protocol.
And while they haven’t gone so far as announce whether or not they will adopt it, the fact they are experimenting with it shows a recognition of its capabilities.
Google’s top concerns about IndexNow seem to be related to sustainability and efficiency.
Google is currently using HTTP/2, an efficient fundamental data transfer protocol for more than half of all crawls, and may ultimately decide this is a better fit for their needs.
Or they may decide to build their own alternative if they determine an API-based approach is needed – and it’s not a foregone conclusion that they will.
But not everyone is on board.
There are plenty of people in the SEO community who don’t see the need for this protocol, instead claiming XML sitemaps already accomplish what IndexNow is trying to do.
You can read more about the debate here.
How Do You Use IndexNow?
Notifying search engines of updates to your web content using IndexNow is a straightforward process:
- Generate an API Key – This is submitted alongside URLs to ensure ownership of the domain. You can do this using an online key generator tool like the one found here.
- Host API Key – Your API key is hosted on the root directory in txt format.
- Submit URLs with Parameters – You can either submit URLs individually or in bulk. Submit your key location as a parameter.
- Check Your Submissions – Using the Bing Webmaster Tools portal, you can verify which URLs have been submitted and discovered.
For examples and more in-depth instructions, you can visit the IndexNow site.
More advanced users can use Python for bulk indexation and URL submission automation.
This requires a basic understanding of Python syntax and familiarity with Python libraries and packages.
We have a handy step-by-step guide to walk you through this process here.
Which Tools Sync With IndexNow?
Any website developer can use the process outlined here to take advantage of IndexNow, but what’s really exciting is that you don’t have to know HTML from HDMI to reap the benefits.
A number of large companies, including LinkedIn, MSN, and GitHub are all planning migrations to IndexNow.
Additionally, other search solution providers like Botify, OnCrawl, Onely and Yext are in the process of adopting this protocol.
Content management systems (CMS) are also jumping on board.
WordPress already offers an IndexNow plugin. Duda supports this proactive approach to web crawling, and Wix plans to integrate soon.
Bing recently announced Rank Math and All In One SEO integration for IndexNow, as well.
Content delivery networks (CDN) supporting IndexNow include Cloudflare and Akamai.
Through their proxy servers, this will speed up data delivery and bring it closer to users.
What Are The Benefits Of Using IndexNow?
In addition to emphasizing the evolution of indexing from pull to push, the primary benefit of IndexNow for publishers is the elimination of the time between updates being made and search engines discovering them.
By allowing webmasters to notify all participating URLs with a single API call, search engine content discovery is streamlined, which in turn makes the web more efficient.
This is advantageous to site owners, as growing requirements are reduced on the server.
Search engine spiders no longer have to perform exploratory crawls to determine whether a page has been updated.
This enables servers to work more efficiently without the need for a redundant site load, which in turn reduces energy usage.
And unlike an XML sitemap, which only redirects SEO crawlers to pages within your website, IndexNow allows you to inform search engines of non-200 status code pages.
IndexNow Frequently Asked Questions
What Happens After I Submit A URL?
IndexNow conveys your changes to search engines, which in turn apply them to page rankings.
Using IndexNow doesn’t guarantee web pages will be crawled or index and it may take time for changes to be reflected.
How Many URLs Can I submit Using IndexNow?
You can provide search engines with a list of up to 10,000 URLs with a single API call.
Will I Rank Better In Search Results If I Index My Content Faster?
Yes, the sooner your pages are indexed, the more quickly they can start competing for high-ranking search engine spots.
What Does All This Mean?
The march of technology is inexorable.
And as such a vital part of the modern world, it’s only natural that changes to the internet, and in particular search engines, would be part of them.
By shifting the process of indexing sites away from a crawling process to something faster and more efficient, IndexNow seems to be on the cusp of a revolution.
And the fact that companies like MSN, GitHub, and WordPress are all joining in seems to hint that this protocol will soon become the norm.
More resources:
Featured Image: ra2 studio/Shutterstock