Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

4
votes
1answer
629 views

How do I deal with content scrapers? [duplicate]

Possible Duplicate: How to protect SHTML pages from crawlers/spiders/scrapers? My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling ...
4
votes
1answer
379 views

Should I block search engines with robots.txt on my Facebook app?

There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
3
votes
3answers
6k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
3
votes
9answers
7k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
3
votes
5answers
2k views

Disallow robots.txt from being accessed in a browser but still accessible by spiders?

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
3
votes
2answers
229 views

How can I allow search engines to index my invite only website in ruby on rails?

I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication: before_filter ...
3
votes
3answers
124 views

Do crawlers require a robots.txt file before indexing content?

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
3
votes
3answers
100 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
3
votes
1answer
662 views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
3
votes
6answers
147 views

Stopping Google index some web pages I have

I have some web pages that google has indexed. Now I added a robots.txt file to exclude them but should I also add a “noindex” tag and if so then how can I best add this?
3
votes
3answers
498 views

Google is indexing unintended content that is either unpublished or a secret part of site

Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M. How is that even possible? Only thing I can ...
3
votes
3answers
437 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...
3
votes
2answers
99 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
3
votes
2answers
194 views

Grapeshot crawler ignoring robots.txt

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
3
votes
3answers
133 views

SEO consideration for duplicate sites

I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...
3
votes
1answer
393 views

How do I disallow robots to access links on a certain port?

I have a private web server set up on a non-conventional port (say 6677). All people can have access to the whole site's urls from 6677. The links by mysite:6677 are also indexed by Google, which is ...
3
votes
2answers
6k views

What's the proper way to handle Allow and Disallow in robots.txt?

I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
3
votes
1answer
447 views

Why does my sitemap have a “.gz” extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
3
votes
2answers
587 views

How can an exception be created for Facebook on robots.txt?

I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
3
votes
1answer
490 views

Why am I getting this message “Some important page is blocked by robots.txt”?

My site's URL is: www.hackinguniverse.org From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt". My robots.txt is: User-agent: ...
3
votes
1answer
3k views

Search engines that do _not_ respect robots.txt?

Are there any search engines that do not care about robots.txt?
3
votes
2answers
221 views

Should I disallow robots on A/B test pages?

I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php. Should I be putting in rules to my ...
3
votes
3answers
289 views

Is my robots.txt working as it should?

I want crawlers to have access to http://www.example.com but not http://www.example.com/ My robots.txt is as follows: User-agent: * Allow: /$ Disallow: / My site is in Google search results, but I ...
3
votes
2answers
116 views

How should “AdSense and DoubleClick” phrase in Google Webmaster Guidelines be interpreted?

I was reading Google Webmaster Guidelines and came across a phrase I do not understand. Make reasonable efforts to ensure that advertisements do not affect search engine rankings. For example, ...
3
votes
1answer
527 views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...
3
votes
1answer
228 views

How to block FeedReader from fetching my content to their site?

As you all can see from the picture below, my site's content is duplicated by FeedReader and indexed at Google. When I clicked at the FeedReader link, it uses some sort of iFrame to draw content from ...
3
votes
1answer
1k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
1answer
416 views

Quick Question, robots.txt Disallow: /*/ does what exactly? [closed]

A SEO firm suggested changing the robots.txt to: User-agent: * Disallow: /*/ Allow: /ims/ I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
3
votes
1answer
240 views

Blocking pages from being crawled with specific URL encoded parameters

While seeing the Google Webmaster Tools report there are certain URL encoded parameters like %5c and %22 coming up in URL's of site. We tried to identify the issue and observed that due to incorrect ...
3
votes
1answer
105 views

Handling SEO for Infinite pages that cause external slow API calls

I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
2
votes
5answers
780 views

Optimizing and securing robots.txt

I have a couple of doubts / questions / ideas related to robots.txt: Can we deny website for all bots except for chosen ones in order to tell other bots not to crawl site: User-agent: * Disallow: / ...
2
votes
2answers
2k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
2
votes
2answers
245 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
2
votes
2answers
104 views

robots.txt: block all webpages except a few number of webpages?

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all the crawlers? User-agent: * Disallow: / Allow: /$ Allow: /a/$ Allow: ...
2
votes
2answers
9k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
2
votes
2answers
78 views

What does “Allow: /” mean in a robots.txt file?

Please help me to understand the following code in robots.txt: Allow: /
2
votes
2answers
306 views

How much Google takes to deindex pages after disallowed from robots?

By accident I have let Google index lots of junk pages. Now I have added them to disallow in robots.txt. Will Google completely remove the pages after seeing the are disallowed from robots? Please ...
2
votes
2answers
517 views

Network unreachable: robots.txt unreachable

I am trying to add a valid sitemap to Google Webmaster. Yet, it says: Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at ...
2
votes
2answers
250 views

robots.txt and disallow

Which one is correct? User-agent: * Disallow: index.php or User-agent: * Disallow: /index.php inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
2
votes
3answers
51 views

Would a self-signed certificate impact my site for search engines?

Would a self-signed certificate impact my site for search engines? By that I mean, do robots refuse to index sites with self-signed certificates?
2
votes
2answers
68 views

Sitemap specified in robots.txt not found by Google

I would like Google to find my sitemap.xml via robots.txt, rather than having to manually upload it. My robots.txt looks like this: User-agent: * Disallow: Sitemap: ...
2
votes
1answer
116 views

Robots.txt to block a parameter instead of a directory

It's my understanding that URLs are of the format example.com/something/somethingelse. It's my understanding that parameters follow the URL with a question mark example.com?l=fr_FR My CMS website ...
2
votes
4answers
843 views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
2
votes
1answer
244 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
2
votes
2answers
758 views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
2
votes
2answers
300 views

Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?

Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
2
votes
1answer
2k views

Google Webmasters Tools: Denied by robots.txt

I have a website: http://46.105.30.3/ And I'm trying to get testing it using the Fetch as Googlebot option in the Google Webmasters Tools. When I try it it returns Denied by robots.txt. My ...
2
votes
2answers
202 views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
2
votes
2answers
634 views

Correct usage of robots.txt file (“Googlebot-Image” and “Mediapartners-Google”)

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress ...