Website Managers Battling with Low Google Crawl Rates

November 2021 has seen many website owners, digital agencies and managers frustrated at the low crawl rates, with both new and refresh content. It took some time, but John Mueller eventually admitted there had been a bug in the Google crawl system that had reduced or prevented Google crawling many websites.

This lapse apparently occurred around the first two weeks of November 2021, but many still think there are still issues out there. Even as this is being written well into December, the subject is still being brought up within live podcasts and question forums.

Personally, I have seen at least 3 of my sites I control severely hit by lower crawl quotas, or maybe this bug in the system, is still there. And Google do not want to admit it.

The Process Of Updating Websites

Unfortunately, I work to a system, where I just get those new sites out there, then when live I continuously add new content and adjust and improve existing content until a decent traffic flow is achieved. This month it has proved more difficult, as the many of the improvements to these websites have not yet been recognised, in some case well over a month after the “onpage” change.

To be fair one site was created with a bulk page creator in WordPress, just to get the pages created and then more and more unique content is added gradually. The site as it stands, does contain lower quality content, that is similar throughout many pages. But in the past, that related to lower ranking positions rather than the pages not being crawled at all. I may have to consider not releasing sites to be live until they have better content, irrelevant of the timescales.

It is not as though Google visits the pages, then they are rejected, within the server logs, there are just little reference to the Google crawl bot visiting the website at all.

As an experiment with a third website, I trialled using a service to improve the CTR (click Through Rates), to see if my crawl quotas would increase. After all this is one of two ways to get Google to crawl more. Either make your site more popular or gain better backlinks. See the results here – using CTR to improve crawl rates.

Google are introducing a new single scroll mobile version of their mobile search results, so maybe they need to slow down crawling to make this new release successful, although they have never needed to do this before.

So until we know better, I am just adding more and more new content in the hope, eventually there is a massive increase in crawling and we get back to normal.