Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

3
votes
9answers
12k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
2
votes
1answer
411 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
252 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
5
votes
1answer
678 views

Do I reference all my *.xml.gz sitemap files in robots.txt

I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files. My question is: ...
2
votes
2answers
288 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
7
votes
4answers
14k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
4
votes
1answer
656 views

How do I deal with content scrapers? [duplicate]

Possible Duplicate: How to protect SHTML pages from crawlers/spiders/scrapers? My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling ...
2
votes
1answer
286 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
0
votes
1answer
126 views

Hiding a particulat page from search engines not to index [duplicate]

Possible Duplicate: Blocking Search Engines to Crawl Parts of a Website? I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
0
votes
1answer
387 views

Is “robots.txt” necessarily needed? [duplicate]

Possible Duplicate: Do I need robots.txt? From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
1
vote
2answers
461 views

How do I have Google not index any of my site, except my domain?

Disallow: / I want Google to index the index page, but nothing else. Will this achieve it?
2
votes
2answers
1k views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
0
votes
3answers
146 views

Robots are still disallowed even without a robots.txt file

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has ...
3
votes
3answers
128 views

Do crawlers require a robots.txt file before indexing content?

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
3
votes
1answer
600 views

Why am I getting this message “Some important page is blocked by robots.txt”?

My site's URL is: www.hackinguniverse.org From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt". My robots.txt is: User-agent: ...
11
votes
2answers
3k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
0
votes
1answer
183 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion) I am building other sites, ...
2
votes
2answers
397 views

Image Hot-links & Google's Instant Previews

I'm trying to adjust my site: 1. stop Image Hot-links 2. block Googlebot-Image (because I do not want my images to be searchable) 3. allow Google's Instant Previews I've tried this but the Instant ...
0
votes
2answers
2k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
2
votes
2answers
169 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
0
votes
2answers
539 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
30
votes
4answers
3k views

Files I should have on every website

I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs. So far I have the ...
7
votes
2answers
168 views

Robots.txt Issue

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following: # go away ...
1
vote
2answers
302 views

robots.txt ignored

I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories. By default it had Disallow: ...
1
vote
2answers
522 views

Robots.txt not working because my site is still searchable

I put a robots.txt in my server This is the code that I putted in User-agent: * Disallow: / I just want my entire site won't be searchable in google. I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
2answers
536 views

Should a mobile-only site use robots.txt?

I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
1
vote
6answers
95 views

Uploading a non-finished website

I have a pretty basic question. I developed a neat little website which I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
2
votes
2answers
4k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
0
votes
1answer
779 views

Restricting crawler activity to certain directories with robots.txt

I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers. In my robots.txt, I have ...
2
votes
2answers
132 views

Robots.txt syntax -> Can this be done?

When creating a robots.txt, are the following the same? # Slurp User-agent: Slurp Disallow: / and # Slurp User-agent: Slurp Disallow: / It seems like for someone disallowing a large number of ...
1
vote
2answers
120 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
2
votes
2answers
330 views

Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?

Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
1
vote
2answers
408 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
0
votes
2answers
146 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
2
votes
2answers
141 views

Pointing to Google that a specific website is for mobiles

I have 2 websites: let's call them A and B. A is a Desktop optimized website. B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A How ...
1
vote
1answer
807 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
5
votes
1answer
198 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
3
votes
2answers
13k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
1
vote
2answers
177 views

Robots.txt and specific file inclusion

Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
4
votes
3answers
1k views

nofollow doesn't stop crawling?

In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" ...
2
votes
1answer
120 views

Adsense and blocking robots from web pages

If I use a meta tag to make a page 'noindex' does this mean that my adsense ads will not be targeted?
1
vote
5answers
475 views

Should I add a “nofollow” attribute to download links, or disallow the URLs in robots.txt?

I have a download link very similar to Opera's one - it's just a script that sends the file. It doesn't have an extension and there's no obvious way to tell that it's actually a download link. So ...
1
vote
2answers
134 views

Problem with my robots.txt

I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
2answers
115 views

Where should the robots.txt file be?

If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ should the robots.txt file be in: subdir.a.com/robots.txt or in a.com/myfolder/robots.txt
6
votes
3answers
1k views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
1answer
558 views

Robots.txt practices with .htaccess redirections (inherits)

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
78 views

Search and Tags for robots.txt

After setting up a blog at blogspot.com I added the site to my webmasters tools of google. I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
5
votes
2answers
2k views

How do you disallow root in robots.txt, but allow a subdirectory?

Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?
3
votes
1answer
5k views

How do I disallow an entire directory with robots.txt?

I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example http://someurl.com/first_url ...