Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

3
votes
4answers
2k views

Google Indexing Pages with noindex meta tag

I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs. For some of the pages listed, there was nothing to ...
3
votes
3answers
809 views

Google is indexing unintended content that is either unpublished or a secret part of site

Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M. How is that even possible? Only thing I can ...
3
votes
3answers
510 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...
3
votes
2answers
48 views

Google can't access non-existent robots.txt

I set up a website a few weeks ago and I'm trying to get Google to crawl it. When logging into Google's Search Console (Webmaster Tools) and within: Crawl > Crawl Errors It reports: Google ...
3
votes
1answer
235 views

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain?

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain? For example: I have example.com and mysubdomain.example.com. mysite.com already has a ...
3
votes
2answers
641 views

Grapeshot crawler ignoring robots.txt

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
3
votes
2answers
3k views

How can an exception be created for Facebook on robots.txt?

I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
3
votes
3answers
139 views

SEO consideration for duplicate sites

I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...
3
votes
1answer
819 views

How do I disallow robots to access links on a certain port?

I have a private web server set up on a non-conventional port (say 6677). All people can have access to the whole site's urls from 6677. The links by mysite:6677 are also indexed by Google, which is ...
3
votes
2answers
532 views

How should “AdSense and DoubleClick” phrase in Google Webmaster Guidelines be interpreted?

I was reading Google Webmaster Guidelines and came across a phrase I do not understand. Make reasonable efforts to ensure that advertisements do not affect search engine rankings. For example, ...
3
votes
2answers
2k views

Why does my sitemap have a “.gz” extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
3
votes
1answer
1k views

How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?

I have this line: Crawl-delay: 50 in the robots.txt file at the root of my website. But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes. I ...
3
votes
1answer
6k views

Search engines that do _not_ respect robots.txt?

Are there any search engines that do not care about robots.txt?
3
votes
2answers
292 views

Should I disallow robots on A/B test pages?

I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php. Should I be putting in rules to my ...
3
votes
3answers
361 views

Is my robots.txt working as it should?

I want crawlers to have access to http://www.example.com but not http://www.example.com/ My robots.txt is as follows: User-agent: * Allow: /$ Disallow: / My site is in Google search results, but I ...
3
votes
1answer
30 views

cookies and useragent is not enough ? to emulate browser?

I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt Let's take an example, I need to get this page: ...
3
votes
1answer
61 views

Point Domain Names With keyword to main website?

This is Question has been asked several time on Stack Exchange. I have read all those answer, still there are certain some technical specification answered. After lot of research on Internet , I ...
3
votes
2answers
163 views

Serving images on CDN; allow crawl of origin or CDN or both?

I have searched and read through a lot of questions regarding this subject, but cannot seem to find anyone that directly addresses my problem. I have a CDN for all my static images img.domain.net, ...
3
votes
1answer
850 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
3
votes
1answer
207 views

Should I disallow(robots.txt) archive/author pages with links already available on the front page?

I am working on a simple Wordpress blog where when an article is published, it appears on ALL these pages: Homepage - Headline(clickable) + 3-line summary Parent category page - Headline(clickable) ...
3
votes
1answer
107 views

Robots.txt destroyed my ranking?

I had to do a massive URL change stuff (Categories and Products) on my e-commerce site while keeping 301 redirect of old URLs to new ones. I did change category URLs (appr: 800 URLs) to new and ...
3
votes
1answer
2k views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...
3
votes
2answers
1k views

Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?

As I understand, having repetitive content is a detriment to search engine placement. Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
3
votes
1answer
253 views

How to block FeedReader from fetching my content to their site?

As you all can see from the picture below, my site's content is duplicated by FeedReader and indexed at Google. When I clicked at the FeedReader link, it uses some sort of iFrame to draw content from ...
3
votes
1answer
2k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
1answer
559 views

Quick Question, robots.txt Disallow: /*/ does what exactly? [closed]

A SEO firm suggested changing the robots.txt to: User-agent: * Disallow: /*/ Allow: /ims/ I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
3
votes
2answers
5k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
3
votes
2answers
41 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
3
votes
1answer
57 views

How do I disallow variable parent directories and subdirectory with robots.txt?

I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages: /category/name1/page/1 /category/name1/page/2 /category/etc/page/3 But I want to allow ...
3
votes
2answers
334 views

I don't want my site to be analyzed on WooRank or builtwith.com

I don't want my site to be analyzed on WooRank or builtwith.com. Is there any way I can do that by editing the robots.txt file or any other possible way?
3
votes
1answer
444 views

Blocking pages from being crawled with specific URL encoded parameters

While seeing the Google Webmaster Tools report there are certain URL encoded parameters like %5c and %22 coming up in URL's of site. We tried to identify the issue and observed that due to incorrect ...
3
votes
2answers
473 views

Will duplicate international (i18n) content hinder SEO rankings?

Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
3
votes
1answer
293 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
3
votes
2answers
54 views

Should I block an .htm resource in robots.txt

We use scripts to create static .htm files to show content on our site. Mainly in an effort to reduce database calls. For example, a script that builds a file every 5 minutes that contains a list of ...
2
votes
5answers
1k views

Optimizing and securing robots.txt

I have a couple of doubts / questions / ideas related to robots.txt: Can we deny website for all bots except for chosen ones in order to tell other bots not to crawl site: User-agent: * Disallow: / ...
2
votes
2answers
3k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
2
votes
2answers
306 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
2
votes
5answers
12k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
2
votes
2answers
186 views

How to add favicon to robots.txt

How can I disallow this URL: http://www.example.com/assets/ico/favicon.png? Would these lines in my robots.txt work? User-agent: * Disallow: /blah/assets* Disallow: /blah/assets/ico/
2
votes
2answers
175 views

robots.txt: block all webpages except a few number of webpages?

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all the crawlers? User-agent: * Disallow: / Allow: /$ Allow: /a/$ Allow: ...
2
votes
2answers
296 views

What does “Allow: /” mean in a robots.txt file?

Please help me to understand the following code in robots.txt: Allow: /
2
votes
1answer
2k views

Strange robots.txt - how and why did it get there?

I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
2
votes
2answers
763 views

How much Google takes to deindex pages after disallowed from robots?

By accident I have let Google index lots of junk pages. Now I have added them to disallow in robots.txt. Will Google completely remove the pages after seeing the are disallowed from robots? Please ...
2
votes
2answers
937 views

Network unreachable: robots.txt unreachable

I am trying to add a valid sitemap to Google Webmaster. Yet, it says: Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at ...
2
votes
2answers
333 views

robots.txt and disallow

Which one is correct? User-agent: * Disallow: index.php or User-agent: * Disallow: /index.php inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
2
votes
2answers
129 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
2
votes
3answers
1k views

Will the already indexed URLs be removed if I use a robots.txt for my site?

Many URLs from my site have been added to Google's index, but a lot of them are outdated and they will never result in a 404 error and take the user to my site's homepage. I have submitted a new ...
2
votes
2answers
216 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
2
votes
1answer
2k views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
2
votes
3answers
11k views

Remove subdomains from Google index and stop indexing them

I am running static content through a CDN, cdn1-cdn5 I am using such subdomains for that. I am loading just images, CSS and JS files this way, but apparently Google has indexed some pages on ...