Google warns against using 403 or 404 status codes for Googlebot crawl rate limiting
Google is warning against using 404 and other 4xx client server status errors, such as 403s, for the purpose of trying to set a crawl rate limit for Googlebot. “Please don’t do that,” Gary Illyes from the Google Search Relations team wrote. Why the notice. There has been a recent increase in the number of
Read More