Google announced that it is sunsetting the search console crawl rate limiter tool, scheduled to be removed on January 8, 2024, citing improvements to crawling that has essentially made it unnecessary.
Search Console Crawl Rate Limiter Tool
The crawl rate limiter tool was introduced to search console fifteen years ago in 2008. The purpose of the tool was to provide publishers a way to control Googlebot crawling so that it didn’t overwhelm the server.
There was a time when some publishers experienced too much crawling, which could result in the server being unable to server webpages to users.
Enough people complained that Google eventually released the tool within search console.
The impact of the tool was to provide Google with data. According to Google, requests to limit crawling typically took about a day to go into effect and remained in effect for 90 days.
Why Google Is Removing Rate Limiter Tool
The announcement stated that crawling algorithms have reached a state where Googlebot can automatically sense when a server is reaching capacity and take immediate action to slow down the crawl rate.
Furthermore, Google stated that the tool was rarely used and when it was used, the crawl rate was generally set to the lowest setting.
Moving forward, the minimum crawl rate will by default be set to a lower rate similar to what publishers tended to request.
According to the announcement:
“With the deprecation of the crawl limiter tool, we’re also setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits.
This means that we effectively continue honoring the settings that some site owners have set in the past if the Search interest is low, and our crawlers don’t waste the site’s bandwidth.”
Making Search Console Less Complex
Removing the tool makes search console easier to use because it’s less cluttered with tools that are rarely used.
This in turn should improve the user experience for search console.
Publishers who continue to have problems with Googlebot’s crawl rate can still use the Googlebot report form to send feedback to Google.
Read Google’s official announcement:
Upcoming deprecation of Crawl Rate Limiter Tool in Search Console
Featured Image by Shutterstock/salarko