3

For a website, GoogleBot daily crawls one particular URL more than one thousand times (mainly between 1,500 and 3,000, sometimes much more).

The Googlebot IP is a real Google one (for example : 66.249.76.21).

This strange (?) behaviour has lasted (at least) for two years. We have recently turn from HTTP to HTTPS, but it is still the same... The first time I met this behaviour, the web page contained a form: the GoogleBot used to send thousands GET and POST requests during a very short period, then stopped suddenly these requests when the form was canceled... but later, it tried again from another page with a form.

When I forbid this URL to the robot, after a short period, another URL is "chosen" by the GoogleBot and crawled again and again (and often in the same period during the day). The last "highly crawled" does not contain any form.

A few days ago, I made another test: the highly crawled URL sends a 302 to the home page.

Results: the GoogleBot follows the 302: but does not follow any files or links from the home page: it sends again and again the same GET request everyday more than 1000 times.

The results is that the main part of the crawl budget is wasted to this single url... Other Robots have not this behaviour.

Have you ever met such a singular observation? Could it be a kind of "security" test?

Any idea about what to do?

  • "requests from this page" - I guess you must mean "to this page"? Are these "thousands of GET and POST requests" identical? Or does the GET/POST data vary? (Although you later state "the same GET request everyday" - is that once a day, or "thousands" of times a day?) Can you post examples of these URLs? – MrWhite Oct 24 '16 at 14:00
  • Hello @w3dk. i send you samples of the log (the url /transparence had a html form). The same day, the number of such requests has sometimes reached more than 9,000 requests ! : 66.249.65.170 www.example.com - [04/Aug/2016:00:00:16 +0200] POST /transparence HTTP/1.1 200 5860 http://www.example.com/transparence Mozilla/5.0 (compatible; Googlebot/2.1) 66.249.65.170 www.example.com - [04/Aug/2016:00:00:17 +0200] GET /transparence HTTP/1.1 200 5784 - Mozilla/5.0 (compatible; Googlebot/2.1) – frederic Laudet Oct 25 '16 at 7:44
  • did you manage to find any clue to your problem above? – Alaa Mar 1 '18 at 10:21
1

If you already have a sitemap for your website, you can try adding that page with a specific changefreq key. For example:

<?xml version="1.0" encoding="UTF-8"?>
<urlset
      xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
            http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url>
  <loc>http://example.com/mypage</loc>
  <changefreq>weekly</changefreq>
  <priority>1.00</priority>
</url>
</urlset>

in the case of your page being http://example.com/mypage. You should submit this sitemap to Google using Google Webmaster Tools, where you can also Change the Googlebot crawl rate.

  • That is a good suggestion, however Google has said that the priority and changreq attributes don't actually change the behavior of the crawler: seroundtable.com/… – Stephen Ostermiller Oct 25 '16 at 15:56
  • Thanks @JorgenPhil for your answer. The site has a sitemap, but a few deep urls are not indicated Inside, including my url ! Last Sunday, almost 4000 requests have been made by the GGBot on the same "overcrawled' url, and a lot of requests (>100) on other deep urls which are not in the sitemap have been also made. I have asked the developper to add these urls. Perhaps your suggestion could resolve this behaviour (which remains abnormal, however). I keep you in touch. – frederic Laudet Oct 26 '16 at 8:26
  • @Stephen Yes, Google decides alone the frequency. Perhaps there is an exceptional bug with GGBot around the absence of deep urls in the sitemap. I will see it very soon if changes are observed by adding thise urls in the sitemap. – frederic Laudet Oct 26 '16 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.