Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

3
votes
1answer
508 views

How do I disallow robots to access links on a certain port?

I have a private web server set up on a non-conventional port (say 6677). All people can have access to the whole site's urls from 6677. The links by mysite:6677 are also indexed by Google, which is ...
1
vote
2answers
207 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
6
votes
3answers
320 views

Does loading images from a server that disallows robots affect SEO?

Lets say a page on example.com has all of its images coming from img.example.com: <img src="http://img.example.com/keyword.jpg" alt="keyword"/> while robots.txt on img.example.com disallows ...
1
vote
2answers
746 views

robots.txt, how effective is it and how long does it take?

We recently updated the site to a single page site using jQuery to slide between "pages". So we now have only index.php. When you search the company on engines such as Google, you get the site and a ...
1
vote
1answer
167 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
1
vote
1answer
268 views

Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm ...
0
votes
1answer
284 views

wordpress robots.txt is this reasonable?

this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms. all the page urls like http://mydomain.com/it/page1, ...
2
votes
1answer
237 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
5
votes
1answer
633 views

Do I reference all my *.xml.gz sitemap files in robots.txt

I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files. My question is: ...
2
votes
1answer
276 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
3
votes
1answer
527 views

Why am I getting this message “Some important page is blocked by robots.txt”?

My site's URL is: www.hackinguniverse.org From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt". My robots.txt is: User-agent: ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
2
votes
2answers
321 views

Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?

Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
0
votes
1answer
366 views

Is “robots.txt” necessarily needed? [duplicate]

Possible Duplicate: Do I need robots.txt? From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
1
vote
2answers
441 views

How do I have Google not index any of my site, except my domain?

Disallow: / I want Google to index the index page, but nothing else. Will this achieve it?
3
votes
2answers
239 views

How can I allow search engines to index my invite only website in ruby on rails?

I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication: before_filter ...
0
votes
3answers
140 views

Robots are still disallowed even without a robots.txt file

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has ...
3
votes
3answers
126 views

Do crawlers require a robots.txt file before indexing content?

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
0
votes
1answer
706 views

Restricting crawler activity to certain directories with robots.txt

I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers. In my robots.txt, I have ...
2
votes
2answers
397 views

Image Hot-links & Google's Instant Previews

I'm trying to adjust my site: 1. stop Image Hot-links 2. block Googlebot-Image (because I do not want my images to be searchable) 3. allow Google's Instant Previews I've tried this but the Instant ...
0
votes
1answer
175 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion) I am building other sites, ...
0
votes
2answers
2k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
2
votes
2answers
160 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
2
votes
1answer
2k views

Google Webmasters Tools: Denied by robots.txt

I have a website: http://46.105.30.3/ And I'm trying to get testing it using the Fetch as Googlebot option in the Google Webmasters Tools. When I try it it returns Denied by robots.txt. My ...
0
votes
1answer
496 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
1
vote
2answers
302 views

robots.txt ignored

I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories. By default it had Disallow: ...
7
votes
2answers
166 views

Robots.txt Issue

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following: # go away ...
1
vote
2answers
498 views

Robots.txt not working because my site is still searchable

I put a robots.txt in my server This is the code that I putted in User-agent: * Disallow: / I just want my entire site won't be searchable in google. I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
2answers
517 views

Should a mobile-only site use robots.txt?

I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
2
votes
2answers
3k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
2
votes
2answers
132 views

Robots.txt syntax -> Can this be done?

When creating a robots.txt, are the following the same? # Slurp User-agent: Slurp Disallow: / and # Slurp User-agent: Slurp Disallow: / It seems like for someone disallowing a large number of ...
4
votes
5answers
110 views

removing robots.txt

Will removing robots.txt make the website 'searchable' again. Curious if you had a robots.txt to avoid indexing, but changed your mind and wanted to get indexed.
2
votes
2answers
135 views

Pointing to Google that a specific website is for mobiles

I have 2 websites: let's call them A and B. A is a Desktop optimized website. B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A How ...
0
votes
2answers
138 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
1
vote
1answer
762 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
5
votes
1answer
154 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
1
vote
2answers
164 views

Robots.txt and specific file inclusion

Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
2
votes
1answer
114 views

Adsense and blocking robots from web pages

If I use a meta tag to make a page 'noindex' does this mean that my adsense ads will not be targeted?
1
vote
2answers
132 views

Problem with my robots.txt

I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
2answers
113 views

Where should the robots.txt file be?

If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ should the robots.txt file be in: subdir.a.com/robots.txt or in a.com/myfolder/robots.txt
4
votes
1answer
936 views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
1answer
128 views

To disallow indexing the category and tag listings in a blog

Mark Wilson says that category and tag listings in a blog should be disallowed in order to prevent duplicate content. I understand this. However, I want to put internal links on keywords in the blog ...
1
vote
1answer
77 views

Search and Tags for robots.txt

After setting up a blog at blogspot.com I added the site to my webmasters tools of google. I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
3
votes
1answer
468 views

Quick Question, robots.txt Disallow: /*/ does what exactly? [closed]

A SEO firm suggested changing the robots.txt to: User-agent: * Disallow: /*/ Allow: /ims/ I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
2
votes
1answer
4k views

How do I disallow an entire directory with robots.txt?

I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example http://someurl.com/first_url ...
1
vote
1answer
58 views

Blocking URLs and canonical questions

We are running a Magento store set up and was looking to block all pages with the exception of a select few. It seems the only way we can do this is with blocking direct paths to files that are in ...
5
votes
1answer
576 views

wget not respecting my robots.txt. Is there an interceptor?

I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little. ...
2
votes
3answers
278 views

After adding *.xls to robots.txt how long will the results continue to show up in google

Is there an estimated amount of time before the document will not appear in google results? We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
3
votes
1answer
4k views

Search engines that do _not_ respect robots.txt?

Are there any search engines that do not care about robots.txt?