November 2021 has seen many website owners, digital agencies, and managers frustrated at the low crawl rates, with both new and refreshed content. It took some time, but John Mueller eventually admitted there had been a bug in the Google crawl system that had reduced or prevented Google from crawling many websites.
This lapse apparently occurred around the first two weeks of November 2021, but many still think there are still issues out there. Even as this is being written well into December, the subject is still being brought up within live podcasts and question forums, it seemd web owners and businesses are still battling with low crawl rates..
Personally, I have seen at least 3 of the sites I control severely hit by lower crawl quotas, or maybe this bug in the system, is still there. And Google does not want to admit it.
The Process Of Updating Websites
Unfortunately, I work to a system, where I just get those new sites out there, then when live I continuously add new content and adjust and improve existing content until a decent traffic flow is achieved. This month it has proved more difficult, as many of the improvements to these websites have not yet been recognised, in some cases well over a month after the “on-page” change.
To be fair one site was created with a bulk page creator in WordPress, just to get the pages created and then more and more unique content was added gradually. The site as it stands, does contain lower-quality content, that is similar throughout many pages. But in the past, that related to lower-ranking positions rather than the pages not being crawled at all. I may have to consider not releasing sites to be live until they have better content, irrelevant to the timescales.
It is not as though Google visits the pages, then they are rejected, within the server logs, there are just a few references to the Google crawl bot visiting the website at all.
As an experiment with a third website, I trialed using a service to improve the CTR (click Through Rates), to see if my crawl quotas would increase. After all, this is one of two ways to get Google to crawl more. Either make your site more popular or gain better backlinks. See the results here – using CTR to improve crawl rates.
Google is introducing a new single-scroll mobile version of their mobile search results, so maybe they need to slow down crawling to make this new release successful, although they have never needed to do this before.
So until we know better, I am just adding more and more new content in the hope, that eventually there will be a massive increase in crawling and we get back to normal.