Blog

There have recently been numerous discussions within the SEO community about crawling and indexing in SEO, particularly about a decline in Googlebot’s crawling activity over the previous weeks. This led some SEO experts to speculate on the real cause of this phenomenon.

One web crawl analysis service creator tweeted an infographic explaining how Google’s crawl rate has been declining since 11 November this year. Although this slowdown does not impact all websites, many people on Reddit and Twitter believe something changed in Google’s way of indexing and proved it with images of activity from Googlebot.

Proof of a Decline in Google’s Crawling

A lot of anecdotal findings regarding Google crawls have been accumulating on social media. However, social media may not be the most accurate source of information as any remark about Google may be expected to receive agreements. Anecdotal evidence, while fascinating, is nothing compared to observations backed up by data.

The founder of a crawling and log analysis service tweeted a graph showing a stunning drop in Google’s crawling activity on 11 November. They said that many big websites were involved in this phenomenon.

The Relationship Between Googlebot Crawling and 304 Server Response Codes

Some users have pointed out that Googlebot no longer crawls pages with a 304 server response code. This type of response code means 304 (Not Modified), anda server generates it when Googlebot or a browser submits a conditional request for a web page.

It means that Googlebot or the browser tells the server that it saved a page in the cache, so there is no need to serve it unless it has been modified or updated.

The HTTP Working Group states that the status code indicates that a conditional HEAD or GET request has been submitted, resulting in a 200 (OK) response if the condition was evaluated to false. This means that the request indicates that the client made the request conditional and has a valid representation already. Therefore, the server doesn’t need to transfer a representation of the target resource and redirects the client to use that stored representation as if it were a part of the 200 response.

Claims That Less Googlebot Crawling is Caused by 304 Response

An SEO stated that they monitored a number of websites with AMP, and a drop occurred on pages with a 304 response. The person who originally tweeted about it posted a graph demonstrating how Google almost stopped crawling web pages with a 304 server response code.

Many SEOs also shared similar experiences where web pages with a 304 response code had a decline in Googlebot crawling. Someone else noticed that their travel pages had a decline in crawls, while their e-commerce web pages had a crawl increase. Others in the SEO community were also sharing screenshots of data from their Google Search Console and Analytics.

Crawling Not Affected  by 304 Response Codes

The Google developer help page on crawling says that a 304 response should not hinder crawling. According to Google’s official documentation, they signal to the indexing pipeline that the page content is similar to how it was when it was previously crawled. The indexing pipeline may change the URLs’ signals, but the response code does not affect it. This statement reassures that the response code should have no impact, but it could also be that Google temporarily changed things or that the developer page isn’t up-to-date.

Cookie Consent Theory

The 304 Server Response theory is one of several theories and approaches to explaining why a web page is not indexed. One person wrote that, after they removed a cookie consent bar, Google began to index more pages. Many SEOs wonder why a cookie response bar would prevent Google from indexing a web page. It could be that the cookie consent bar may trigger a 304 response and cause Google not to index the page.

Reddit Discussions About Reduced Googlebot Crawls

The decline in Googlebot crawls also reached Reddit. A Reddit user recounted how their successful site’s articles were indexed within 10 minutes of uploading them using Google Search Console. However, things changed in November as less than half of their newly-published articles were indexed, even if they had already manually submitted them after publishing.

Many other Reddit users shared a similar issue, and they concluded that there could be something going on with Google.

Mueller Says Nothing Is Broken

Google’s John Mueller also shared insights. He said that at the moment, he doesn’t believe there is anything wrong with Google’s indexing methods. However, he thinks Google is just more critical about what they choose to include in their indexes, just as any search engine should.

The Possibility of Google Testing New Crawling Patterns

In October, Bing revealed the new IndexNow, an open-source indexing protocol that aims to reduce the number of times search engines crawl web pages and the amount of energy consumed at data centres by crawling and powering servers. The new method benefits publishers since it shortens the time to notify search engines when sites are created or modified, resulting in faster indexing of web pages.

In November, Google said it would test the new IndexNow method to see if they could gain any advantages. One of the most pressing problems today is energy conservation and carbon reduction. It may be possible that Google is developing ways to reduce crawling without necessitating a substantial change in protocol.

There are reports that Google has stopped indexing entirely, but this is incorrect. However, there is evidence that Googlebot’s indexing behaviour has been altered in response to the current debate on social media, with screenshots of data to support this matter.

Webmix Networks SEO Can Help You With Crawling And Indexing

Webmix Networks SEO is a US-based SEO company that specialises in optimising web pages, making your website ready for crawling and indexing in SEO. Our services help your website attract more clicks and traffic, bringing more potential clients in for your business.

We are experts in converting site traffic to paying customers. Our affordable SEO packages ensure that you can increase your profits without any upfront costs or contracts.

Denver SEO Services

Our services are tailored to suit your specific business needs, so you can be sure that you’ll get the best results possible. Whether it’s an ecommerce store, blog, content marketing project, or anything else, we can get started on it right away!