0

As per the title, is it possible to block all search engine indexing using DNS? Most guides point towards robots.txt or meta headers but I'd like to investigate the possibility of blocking all indexing for a domain that will host all of our QA sites (which we don't want appearing on search engines).

3

No, you can't block indexing using DNS - unless you made your site inaccessible! If there was a DNS method (like some kind of txt record or something) then the search engine would need to make a secondary lookup to find it.

If you need to block an entire domain, then look at the X-Robots-Tag HTTP response header (HTTP equivalent of the robots meta tag). Although most people will just block crawling with robots.txt (although note that this doesn't necessarily prevent indexing if the blocked resource gets linked to).

  • TXT records was the way I was expecting to be able to do it, if it indeed existed. That X-Robots-Tag header looks interesting, thanks for that info. – James Jan 9 '17 at 9:27
3

No. A DNS is a simple mechanism, you give it a domain, it returns an A-record (with the targetserver's IP address). There isn't really anything you can do.

Even if you manage to create a check in the DNS record, or add a meta tag with robots info, or create a robots.txt, they don't have to listen to it. They're simply being polite and respect the robots.txt, but there is no technical obligation for them to do so.

If they change the HTTP_USER_AGENT or a X-ROBOTS request-header info your check will no longer work. Or when you have an IP check and they switch to a new/unknown IP addrress, your check no longer works.

This is why I prefer to see these are guides for crawlers, not as rules.
Always assume that everything a visitor can see, can been seen by a crawler, one way or another. From a security perspective, this is safer.

  • What do you mean by, "If the crawler just changes its header info, or crawls from a new IP address, your check will not work."? What "check" is that? – MrWhite Jan 9 '17 at 9:23
  • Updated, more clear like this? – Martijn Jan 9 '17 at 9:24
  • Sorry, still not sure what you mean? There shouldn't need to be any "check" that the OP/site needs to make regarding the request? What does it matter if the crawler changes the request headers (to pretend to be a regular visitor)? – MrWhite Jan 9 '17 at 9:35
  • Most robots respect your robots.txt but there is absolutely nothing to stop them if they decide to ignore it (or a meta tag, or a dns-check, or whatever check). Even if you block an IP, if they go to a new/unknown IP address, you can start all over. – Martijn Jan 9 '17 at 9:43
  • Ah OK, I see what you're saying. But yeah, DNS doesn't work like that; it's a simple one-way lookup. But neither would you want/need to perform a conditional check like that (the user-agent either respects the response or not). (What's a X-ROBOTS request-header?) If you did need to go so far as requiring a "check", then it should probably be password protected. – MrWhite Jan 9 '17 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.