1

A web application I wrote is hosted on an in-house server with the name myserver, which is under my university's domain (department.uni.edu), resulting in the server's address being myserver.department.uni.edu. When I Google myserver, the first result is that exact server hosting the web application.

I have a robots.txt file for the application (root directory) with the following contents:

User-agent: *
Disallow: /

It's the actual server domain name that was indexed, and not anything in the web application.

I know that I can remove search results with Google Webmaster Tools, but how do I prevent Google from indexing a server's domain name (or address)? I believe the server is running Nginx on Ubuntu 14.10 (I am not the person in charge of the server, just coding the web application).

The desire here is to prevent the server from being indexed by web searching tools such as Google, Bing, Yahoo, etc. - basically block all known search engine crawlers.

Perhaps a solution is to block all crawlers to the subdomain's root (mysever.department.university.edu) using an Nginx rewrite rule such as:

map $http_user_agent $limit_bots {
     default 0;
     ~*(google|bing|yandex|msnbot) 1;
     ~*(AltaVista|Googlebot|Slurp|BlackWidow|Bot|ChinaClaw|Custo|DISCo|Download|Demon|eCatch|EirGrabber|EmailSiphon|EmailWolf|SuperHTTP|Surfbot|WebWhacker) 1;
     ~*(Express|WebPictures|ExtractorPro|EyeNetIE|FlashGet|GetRight|GetWeb!|Go!Zilla|Go-Ahead-Got-It|GrabNet|Grafula|HMView|Go!Zilla|Go-Ahead-Got-It) 1;
     ~*(rafula|HMView|HTTrack|Stripper|Sucker|Indy|InterGET|Ninja|JetCar|Spider|larbin|LeechFTP|Downloader|tool|Navroad|NearSite|NetAnts|tAkeOut|WWWOFFLE) 1;
     ~*(GrabNet|NetSpider|Vampire|NetZIP|Octopus|Offline|PageGrabber|Foto|pavuk|pcBrowser|RealDownload|ReGet|SiteSnagger|SmartDownload|SuperBot|WebSpider) 1;
     ~*(Teleport|VoidEYE|Collector|WebAuto|WebCopier|WebFetch|WebGo|WebLeacher|WebReaper|WebSauger|eXtractor|Quester|WebStripper|WebZIP|Wget|Widow|Zeus) 1;
     ~*(Twengabot|htmlparser|libwww|Python|perl|urllib|scan|Curl|email|PycURL|Pyth|PyQ|WebCollector|WebCopy|webcraw) 1;
 }

location / {
    if ($limit_bots = 1) {
        return 403;
    }
}

(borrowed from GD Hussle)

but, would this be sufficient or would something more sophisticated be necessary?

Improve this question
5
  • The robots.txt file should do it. Just make sure that it is in the web root and can be read okay. It may be that someone linked to your site? If this is the case, Google will list it with a SERP link only and a notation that the site description is not available. Google should not do this, but they do. Otherwise, you can block access and do a full HTTP request drop instead of a 404 or a 410. This is typically done in a firewall or you can possibly use ModSecurity or other web application firewall.
    – closetnoc
    Commented Jan 25, 2016 at 17:36
  • When you find your server's name does it say "A description for this result is not available because of this site's robots.txt – learn more."? I don't think you can make Google not index the domain name itself.
    – user3919
    Commented Jan 25, 2016 at 17:56
  • @barrycarter This is exactly correct. The problem is that the root URL for the server is where the web application is hosted, meaning though the web app isn't indexed itself, the server subdomain which is indexed is pointing to the web application! Now, the server could rewrite the URL to myserver.department.uni.edu/webapp, and have the root return some other thing, but that would break every existing URL. And we can't break URLs, because redirecting would make it so Google would again point to the web application. Very frustrating...
    – Chris Cirefice
    Commented Jan 25, 2016 at 19:40
  • @closetnoc The robots.txt file is definitely being read correctly (see barrycarter's comment). Since I don't manage the server: just doing basic Googling, would configuring Nginx to respond with an HTTP 444 for the subdomain root URL (if the user agent is a bot) be sufficient to prevent its indexing on search engines? Basically, if Google/other bots crawl myserver.department.uni.edu, I want to drop the connection for them, but allow non-bots to connect successfully.
    – Chris Cirefice
    Commented Jan 25, 2016 at 19:45
  • I just got back in. A 404 error will say that a particular page is Temporarily Gone while a 410 will say that a particular page is Gone. Either should work okay except that a 404 takes longer. That would be fine. But I am not sure that Google will stop trying to access the sub-domain per se' as long as a web server is responding with a response code. A 404 or 410 is on a page by page basis. I was suggesting, and only if it was easy to do, was to make it appear that the server was gone. ;-) Stick with the 404!
    – closetnoc
    Commented Jan 26, 2016 at 4:27

1 Answer 1

Reset to default
2

With robots.txt you can control crawling, not indexing. If a search engine is not allowed to crawl a document on your host, it might still index its URL, e.g. if it found the link on an external site.

You can control indexing with the meta-robots element or the X-Robots-Tag HTTP header (see examples).

You have to decide if you want to allow search engines to crawl but not to index, or to index but not to crawl. Because if you disallow crawling in robots.txt, search engines won’t be able to reach your documents, so they’ll never learn that you don’t want these to get indexed.

Improve this answer
9
  • So if I understand this correctly, I can't possibly drop HTTP requests into the root URL because then crawlers can't read robots.txt, meaning that anything linked in the site would be likely to turn up in search results? So it would essentially be impossible to disallow indexing of the root url?
    – Chris Cirefice
    Commented Jan 29, 2016 at 14:35
  • @ChrisCirefice: Not sure I understand. --- If you don’t want the root URL to get indexed, you could send the HTTP header: X-Robots-Tag: noindex (which, of course, only works for the search engines that support it). For this to work, your robots.txt must allow crawling of the root URL (otherwise bots would never see the HTTP header).
    – unor
    Commented Jan 29, 2016 at 15:08
  • I want to 1. prevent indexing of the site's root URL (the server domain myserver.department.university.edu) and 2. any of the site's contents in myserver.department.university.edu/. I mentioned earlier writing an Nginx rule to drop HTTP requests if the request was to the root URL; but, if I did that then crawlers wouldn't be able to read robots.txt, if I understand your answer. Meaning that I can't do both of my desired things. I am already preventing indexing of anything past /, but I don't want the server's domain to be in search results either. It seems that I can't do both.
    – Chris Cirefice
    Commented Jan 29, 2016 at 15:12
  • @ChrisCirefice: I think we are talking about different things. If you disallow crawling of your site (in robots.txt), you can’t disallow indexing of your site. But if you are fine with bots crawling your site, you can disallow indexing (= bots can look around, but are not allowed to list the URLs in their search results). -- With "dropping", you mean sending a 403 instead of 200, right? But this wouldn’t change anything regarding indexing (while some bots might not want to index 403’d documents, they would still be "allowed" to).
    – unor
    Commented Jan 29, 2016 at 15:25
  • Unor, correct, responding to HTTP requests (based on User-Agent as in my suggested Nginx solution in the question) at the root url of myserver.department.university.edu/. The idea is to not have the server's contents show up in search engine results. Bots can crawl all they want, they won't find any useful resources because they are all protected by login - the server will serve HTTP 403. I don't mind crawling, as long as nothing on the site, including the server's root url, is viewable in search engine results. Does that make sense? Sorry, I lack the necessary terminology!
    – Chris Cirefice
    Commented Jan 29, 2016 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.