1

I'm in a situation where I'm looking to set a different DNS record for a certain file on a domain, (specifically www.domain.com/robots.txt), from the root (www.domain.com).

(The situation is that I've set the web root to point to a non-live record so the website appears to be non-existent but still has mx records to allow email - but I've now realised I actually need robots.txt to be live and accessible for crawlers such as archive.org so that they respect my site's wishes to not show past snapshots of the site.)

Is it possible to do this it in DNS (like some sort of 'TXT' record?), or is there no way to differentiate the DNS of anything trailing .com, from what's already been set for www.domain.com?

The issue is that I really need the web root to solidly appear like a non-existent site, as discussed in my earlier above-linked question. It can't be a 'fake' non-existent page, but an actual one.

Perhaps those two things are simply incompatible?

1

With DNS A records, you cannot set records for a specific file. DNS is all or none for the domain.

If you must serve a robots.txt file, then you have to have DNS resolve to a valid domain and have a web server that will answer that request.

Here is what I recommend:

DNS

Setup a CNAME to point www.domain.com to domain.com. Setup an A record to point domain.com to your server's IP address.

Web Server

On your server (assuming Apache), you can use .htaccess to limit what files are being served.

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{REQUEST_URI} !/robots.txt [NC]
RewriteRule ^(.*) - [L,R=404]

This will return a 404 (or any code you want) for all pages but /robots.txt.

This way you can host robots.txt but not any other pages.

There is also the 410 Gone HTTP response code but I am not sure how bots handle it.

10.4.11 410 Gone

The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) SHOULD be used instead. This response is cacheable unless indicated otherwise.

The 410 response is primarily intended to assist the task of web maintenance by notifying the recipient that the resource is intentionally unavailable and that the server owners desire that remote links to that resource be removed. Such an event is common for limited-time, promotional services and for resources belonging to individuals no longer working at the server's site. It is not necessary to mark all permanently unavailable resources as "gone" or to keep the mark for any length of time -- that is left to the discretion of the server owner.

  • Aha, you've come up with a solution that basically I worked out yesterday and I must say it works great - I can successfully report archive.org is honoring BOTH www.domain and domain blocking, from just the http:// robots.txt! But I have one change which you may want to update your answer with, which IMO works even better - I'm doing just that in terms of CNAME and A records, but in my .htaccess I have THIS code, which means everything BUT domain.com/robots.txt will not merely 404, but redirect to the non-live DNS, which for my needs is superior. See: pastebin.com/6c7Qaqqt – user26888 Jun 18 '14 at 3:56
  • (Just to clarify, my CNAME record for www.domain.com points to a non-live address (and the A record for domain.com to the server IP as normal), so www.domain.com/robots.txt is NOT live, but with a 301 redirect trick for everything on the webserver but with robots.txt as an exception (which is what my pastebin is), I can at least have http://domain.com/robots.txt live, and archive.org, it appears, is honoring it for both non-www AND www.) – user26888 Jun 18 '14 at 4:02
  • Technically the 410 response code would be appropriate but I am not sure how many systems actually use that code. You may want to check access logs to see if crawlers actually honor the 301 or see it as an error and return later to check again. – jeffatrackaid Jun 18 '14 at 19:04
  • I'm not so bothered by crawlers not honoring the 301, but as long as user browsers do (which from my testing it always does), it is fine. And in fact that would only be good if www / robots.txt doesn't 301 to a non-live resource as it does in the browser currently...so luckily, it all appears bullet proof (against whatever results I've checked) for now... 410 sounds interesting, thank you, and if anything turns out to have a hole in its defense in future maybe I can look into it as an idea. Thanks – user26888 Jun 19 '14 at 1:38
0

Your question is a bit confusing. But I think I understand. I am going to make some assumptions. One is, you have a web server enabled, but no website.

**Update: Okay, from the comments, your question is not clear at all. I am updating the answer to be more complete.

No. DNS stands for Domain Name Service. It translates domain names to routable IP addresses. Nothing more.

If you do not want snapshots of an empty site, you do not need a robots.txt file, you simply turn off the web server. By turn off, I mean stopping the service.

Here is the Apache documentation:

http://httpd.apache.org/docs/2.2/en/stopping.html

Here is a Microsoft page (may not be the latest):

http://technet.microsoft.com/en-us/library/cc732317(v=ws.10).aspx

If this is not what you want to do, then simply put up a robots.txt in your root, but make sure that either www redirects to mydomain.com or mydomain.com redirects to www.

# redirect root to www
RewriteCond %{HTTP_HOST} ^mydomain\.com$ [NC]
RewriteRule ^(.*)$ http://www.mydomain.com/$1 [R=301,L]

Archive.org does a very poor job of respecting website owners wishes. There are holes in their processes and policies. For example, if your site disappears, archive snapshots can reappear. As well, it is very possible that archive.org see and obeys robots.txt via www.mydomain.com but not vie mydomain.com. Even when the robots.txt can be read either way, archive.org can ignore these and index your site anyway. Having said that, if all the stars align, archive.org can read the robots.txt file and drop all snapshots, but I have seen them come back months later.

Make sure you can access your robots.txt file from both www.mydomain.com and mydomain.com. This is important. Create the file and put an entry in it like this:

User-agent: ia_archiver
Disallow: /

If you cannot access your site as www.mydomain.com and mydomain.com, check your DNS server for the following:

An A record associating your IP address to mydomain.com A CNAME record associating www.mydomain.com to mydomain.com -or- An A record associating your IP address to www.mydomain.com

I think this should do it. I can update more if it is required.

  • I only have a website in that I have a blank index.html, and a 'disallow all' robots.txt. Right now, archive.org is showing past snapshots of my website despite blocking it earlier this year via robots.txt, and I realise it's because I changed the DNS for www.mydomain.com to point to nothingness, thus breaking that retroactive archive.org robots.txt blocking. Archive.org already recorded the snapshots, so turning off my website won't do anything to block what they already captured. I know I can submit a request to them as well, but I imagine other services fall into the same boat anyway. – user26888 Jun 17 '14 at 9:14
  • @foregon: Editing of robots.txt will not affect ALREADY INDEXED content. – phoops Jun 17 '14 at 12:30
  • @edvinas.me with the archive.org Internet Way back Machine (which as explained is my context), it does. Go check it/test it with a domain of yours to see for yourself, or Google to cross-check my claim. It's unorthodox, but they respect robots.txt for deciding whether to (publicly) show ALREADY INDEXED content, on their Internet Wayback Machine archive. I had no idea they STILL KEEP the content for later if the website STOPS stating 'Disallow: /' in robots.txt, but they do, based on observation with my own website. – user26888 Jun 17 '14 at 12:36
  • @edvinas.me I updated the answer with more info. – closetnoc Jun 17 '14 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy