2

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each subdomain. I only left the subdomains active so that people who accessed the old URLs can be automatically redirected to the proper URLs.

I have removed my subdomains (except for the one active one I'm using) from all webmaster tools accounts and I make zero references to them on my site.

Is there a way to minimize the amount of times search engines scan robots.txt on my obsolete subdomains?

Ideally, I'd like to have the search engines scan the robots.txt found on each obsolete domain only one time, and then scan all the necessary URLs on the subdomain and that's it, not scan robots.txt every few seconds.

  • If this was not such a public forum, I would create a few very new robots.txt file directives for you. ;-) – closetnoc Mar 12 '16 at 0:03
  • "scan robots.txt every few seconds" - really?! Are you sure that's the Googlebot? I wouldn't have removed the subdomains from the Webmaster Tools (Search Console) account(s) - you then have the option to control the crawl rate - although whether that influences how often robots.txt is checked I don't know (I somehow doubt it). – MrWhite Mar 12 '16 at 0:17
  • Officially, the last I knew, Google would check your robots.txt before fetching any page if the robots.txt has not been checked within 24 hours. In otherwords, as frequently as once a day. – closetnoc Mar 12 '16 at 3:24
  • What does a request for your subdomain's robots.txt file return? Is it redirected (301) like everything else? A 200 OK and a valid file? Or a 404, 410, ...? If it's a 410 Gone then I would think the number of requests would lessen over time. (?) – MrWhite Mar 12 '16 at 10:59
  • At the moment, 200 OK and a valid file. – Mike Mar 12 '16 at 22:16
1

May be Google check your robots.txt frequently, because you have moved your site content to new location. There is no any way to control robots.txt crawling rate.

But, If your robots.txt did not blocked anything like any directory, then don't use robots.txt at all. For blocking other bots, you can use httacess. I have seen most of sites, don't use robots.txt properly. They specify useragent and allow them to crawl, like this,

User-agent: Googlebot
Allow: /
User-agent: Mediapartners-Google
Disallow: 

This type of robots.txt is junk, it just waste space and bandwidth.

So Use Robots.txt only, if you really want to block such a thing, For small type of things, I recommended to use X-Robots header tag.

0

You should keep the subdomains setup with redirects to your top level domain - that way old traffic will land on a functional website and Google will re-crawl over time and realised all the traffic has been directed elsewhere

Add this to .htaccess and modify it to suite your sites Url

RewriteEngine on
RewriteCond %{HTTP_HOST} ^m\.example\.com$
RewriteRule ^ http://example.com/m%{REQUEST_URI} [L,P]

Read more here: https://stackoverflow.com/questions/1456956/redirect-subdomain-into-top-level-domains-folder

  • How does this answer the question? The OP states in their first paragraph, "I ... left the subdomains active so that people who accessed the old URLs can be automatically redirected to the proper URLs.". But the code you have posted isn't an external redirect, it's a proxied request, so the search engines will never "realised all the traffic has been directed elsewhere". – MrWhite Mar 12 '16 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.