Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

3
votes
1answer
72 views

Robots.txt destroyed my ranking?

I had to do a massive URL change stuff (Categories and Products) on my e-commerce site while keeping 301 redirect of old URLs to new ones. I did change category URLs (appr: 800 URLs) to new and ...
3
votes
2answers
329 views

How should “AdSense and DoubleClick” phrase in Google Webmaster Guidelines be interpreted?

I was reading Google Webmaster Guidelines and came across a phrase I do not understand. Make reasonable efforts to ensure that advertisements do not affect search engine rankings. For example, ...
3
votes
1answer
1k views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...
3
votes
1answer
239 views

How to block FeedReader from fetching my content to their site?

As you all can see from the picture below, my site's content is duplicated by FeedReader and indexed at Google. When I clicked at the FeedReader link, it uses some sort of iFrame to draw content from ...
3
votes
1answer
1k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
1answer
503 views

Quick Question, robots.txt Disallow: /*/ does what exactly? [closed]

A SEO firm suggested changing the robots.txt to: User-agent: * Disallow: /*/ Allow: /ims/ I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
3
votes
2answers
135 views

I don't want my site to be analyzed on WooRank or builtwith.com

I don't want my site to be analyzed on WooRank or builtwith.com. Is there any way I can do that by editing the robots.txt file or any other possible way?
3
votes
1answer
354 views

Blocking pages from being crawled with specific URL encoded parameters

While seeing the Google Webmaster Tools report there are certain URL encoded parameters like %5c and %22 coming up in URL's of site. We tried to identify the issue and observed that due to incorrect ...
3
votes
2answers
355 views

Will duplicate international (i18n) content hinder SEO rankings?

Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
2
votes
5answers
1k views

Optimizing and securing robots.txt

I have a couple of doubts / questions / ideas related to robots.txt: Can we deny website for all bots except for chosen ones in order to tell other bots not to crawl site: User-agent: * Disallow: / ...
2
votes
2answers
2k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
2
votes
2answers
272 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
2
votes
3answers
87 views

How to add favicon to robots.txt

How can I disallow this URL: http://www.example.com/assets/ico/favicon.png? Would these lines in my robots.txt work? User-agent: * Disallow: /blah/assets* Disallow: /blah/assets/ico/
2
votes
2answers
131 views

robots.txt: block all webpages except a few number of webpages?

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all the crawlers? User-agent: * Disallow: / Allow: /$ Allow: /a/$ Allow: ...
2
votes
1answer
133 views

Is it possible to exclude Data-URIs (data:image/xxx) through robots.txt?

At the moment Google (I can not find similar informations for Bing, Yahoo!, etc.) does not index Data-URI. But until it is possible I like to know if it is possible to exclude Data-URIs of being ...
2
votes
2answers
163 views

What does “Allow: /” mean in a robots.txt file?

Please help me to understand the following code in robots.txt: Allow: /
2
votes
1answer
2k views

Strange robots.txt - how and why did it get there?

I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
2
votes
2answers
580 views

How much Google takes to deindex pages after disallowed from robots?

By accident I have let Google index lots of junk pages. Now I have added them to disallow in robots.txt. Will Google completely remove the pages after seeing the are disallowed from robots? Please ...
2
votes
2answers
630 views

Network unreachable: robots.txt unreachable

I am trying to add a valid sitemap to Google Webmaster. Yet, it says: Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at ...
2
votes
2answers
302 views

robots.txt and disallow

Which one is correct? User-agent: * Disallow: index.php or User-agent: * Disallow: /index.php inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
2
votes
3answers
359 views

Will the already indexed URLs be removed if I use a robots.txt for my site?

Many URLs from my site have been added to Google's index, but a lot of them are outdated and they will never result in a 404 error and take the user to my site's homepage. I have submitted a new ...
2
votes
2answers
130 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
2
votes
3answers
225 views

Would a self-signed certificate impact my site for search engines?

Would a self-signed certificate impact my site for search engines? By that I mean, do robots refuse to index sites with self-signed certificates?
2
votes
1answer
140 views

Robots.txt to block a parameter instead of a directory

It's my understanding that URLs are of the format example.com/something/somethingelse. It's my understanding that parameters follow the URL with a question mark example.com?l=fr_FR My CMS website ...
2
votes
3answers
768 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
2
votes
1answer
286 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
2
votes
2answers
1k views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
2
votes
2answers
330 views

Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?

Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
2
votes
1answer
2k views

Google Webmasters Tools: Denied by robots.txt

I have a website: http://46.105.30.3/ And I'm trying to get testing it using the Fetch as Googlebot option in the Google Webmasters Tools. When I try it it returns Denied by robots.txt. My ...
2
votes
3answers
490 views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
2
votes
1answer
40 views

should you let search result page to be indexed

I have a pages of a catalogs, which contains a list of keywords of all products inside that catalogs, and I want to create a link from those keywords to a search page. For example, if the page ...
2
votes
1answer
91 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
2
votes
2answers
1k views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
2
votes
2answers
1k views

Correct usage of robots.txt file (“Googlebot-Image” and “Mediapartners-Google”)

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress ...
2
votes
3answers
296 views

Hiding php includes from search spiders?

Quick and simple question. I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are ...
2
votes
2answers
397 views

Image Hot-links & Google's Instant Previews

I'm trying to adjust my site: 1. stop Image Hot-links 2. block Googlebot-Image (because I do not want my images to be searchable) 3. allow Google's Instant Previews I've tried this but the Instant ...
2
votes
2answers
169 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
2
votes
2answers
4k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
2
votes
2answers
141 views

Pointing to Google that a specific website is for mobiles

I have 2 websites: let's call them A and B. A is a Desktop optimized website. B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A How ...
2
votes
1answer
120 views

Adsense and blocking robots from web pages

If I use a meta tag to make a page 'noindex' does this mean that my adsense ads will not be targeted?
2
votes
3answers
282 views

After adding *.xls to robots.txt how long will the results continue to show up in google

Is there an estimated amount of time before the document will not appear in google results? We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
2
votes
1answer
63 views

Robots.txt vs. Noindex Tags

According to matt cutts, even though we block the page using Robots.txt , it is better using noindex tag. ( Source : https://www.youtube.com/watch?v=KBdEwpRQRD0) If it is blocked by Robots.txt, how ...
2
votes
1answer
43 views

Is wildcard in Robots.txt in middle of string recognized?

I need some string for robots.txt like: disallow: /article/*/ but I don't know if this is a proper way to do this or not?! I need, for example, the following URLs to be followed: ...
2
votes
2answers
102 views

Does the Instant Preview in Google Webmaster Tools take robots.txt in account?

Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as googlebot, it fetches ok, ...
2
votes
2answers
49 views

Does Google crawl page when G+ logo is embedded?

I have a robots.txt file in the root to prevent search engines indexing any of my sites. I have a social media management application embedded on my site that includes all major social media channels ...
2
votes
3answers
224 views

Is it safe to block redirected (but still linked) URLs with robots.txt?

I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout the site the unclean URLs are linked in menus, content, products, etc. ...
2
votes
1answer
110 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
2
votes
1answer
68 views

To allow or disallow alphabetical content directory pages?

We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
2
votes
1answer
665 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...