Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

6
votes
3answers
319 views

Does loading images from a server that disallows robots affect SEO?

Lets say a page on example.com has all of its images coming from img.example.com: <img src="http://img.example.com/keyword.jpg" alt="keyword"/> while robots.txt on img.example.com disallows ...
1
vote
2answers
686 views

robots.txt, how effective is it and how long does it take?

We recently updated the site to a single page site using jQuery to slide between "pages". So we now have only index.php. When you search the company on engines such as Google, you get the site and a ...
1
vote
1answer
157 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
4
votes
3answers
290 views

Bad Bot blocking Revisited

I've read a lot about bad bot blocking, php scripts, .htaccess techniques, etc... Is this a valid method? Since .htacces can rewrite and send a bad bot a 403 deny or forward to something like spam ...
1
vote
2answers
173 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
6
votes
4answers
5k views

In Robots.txt, how do I disallow all pages except for the index page?

I want google to index my index page, but do not index any other pages. User-agent: * Disallow: / So far I have this. However...when I do this, Google does not index my index page. When I search ...
1
vote
1answer
528 views

Complex Disallow pattern in robots.txt

I have a URL like this: www.example.com/freelance-jobs-new-york I had a problem and many duplicated pages have been created like this: www.example.com/freelance-jobs-new-york-php-php ...
1
vote
1answer
268 views

Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm ...
1
vote
1answer
455 views

Block access to robots.txt file .htaccess

I read in this forum that this .htaccess code should block access to the robot.txt file: RewriteCond %{HTTP_USER_AGENT} ^Mozilla RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely) RewriteRule ...
0
votes
0answers
109 views

Disallow age gate/verification page in robots.txt?

Should I disallow the age gate page in robots.txt? I need to make sure the crawlers can get through the gate.
1
vote
2answers
571 views

How to submit sitemap when your website has partial https? - Error: “Not in Domain”

My website is an eCommerce that is set up to do HTTP for the item browsing portion, but HTTPS for things like shopping cart, contact us, etc.. (anything that has forms on it). I've submitted my ...
0
votes
1answer
283 views

wordpress robots.txt is this reasonable?

this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms. all the page urls like http://mydomain.com/it/page1, ...
5
votes
3answers
8k views

How to remove old robots.txt from google as old file block the whole site [duplicate]

I have a website which still shows old robots.txt in the google webmaster tools. User-agent: * Disallow: / Which is blocking Googlebot. I have removed old file updated new robots.txt file with ...
3
votes
9answers
9k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
2
votes
1answer
390 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
233 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
5
votes
1answer
612 views

Do I reference all my *.xml.gz sitemap files in robots.txt

I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files. My question is: ...
2
votes
2answers
268 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
7
votes
4answers
13k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
4
votes
1answer
643 views

How do I deal with content scrapers? [duplicate]

Possible Duplicate: How to protect SHTML pages from crawlers/spiders/scrapers? My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling ...
2
votes
1answer
270 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
0
votes
1answer
122 views

Hiding a particulat page from search engines not to index [duplicate]

Possible Duplicate: Blocking Search Engines to Crawl Parts of a Website? I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
0
votes
1answer
359 views

Is “robots.txt” necessarily needed? [duplicate]

Possible Duplicate: Do I need robots.txt? From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
1
vote
2answers
429 views

How do I have Google not index any of my site, except my domain?

Disallow: / I want Google to index the index page, but nothing else. Will this achieve it?
2
votes
2answers
900 views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
0
votes
3answers
135 views

Robots are still disallowed even without a robots.txt file

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has ...
3
votes
3answers
125 views

Do crawlers require a robots.txt file before indexing content?

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
3
votes
1answer
520 views

Why am I getting this message “Some important page is blocked by robots.txt”?

My site's URL is: www.hackinguniverse.org From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt". My robots.txt is: User-agent: ...
8
votes
2answers
3k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
0
votes
1answer
173 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion) I am building other sites, ...
2
votes
2answers
394 views

Image Hot-links & Google's Instant Previews

I'm trying to adjust my site: 1. stop Image Hot-links 2. block Googlebot-Image (because I do not want my images to be searchable) 3. allow Google's Instant Previews I've tried this but the Instant ...
0
votes
2answers
2k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
2
votes
2answers
156 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
0
votes
1answer
482 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
28
votes
3answers
3k views

Files I should have on every website

I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs. So far I have the ...
7
votes
2answers
165 views

Robots.txt Issue

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following: # go away ...
1
vote
2answers
300 views

robots.txt ignored

I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories. By default it had Disallow: ...
1
vote
2answers
483 views

Robots.txt not working because my site is still searchable

I put a robots.txt in my server This is the code that I putted in User-agent: * Disallow: / I just want my entire site won't be searchable in google. I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
2answers
514 views

Should a mobile-only site use robots.txt?

I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
1
vote
6answers
93 views

Uploading a non-finished website

I have a pretty basic question. I developed a neat little website which I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
2
votes
2answers
3k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
0
votes
1answer
681 views

Restricting crawler activity to certain directories with robots.txt

I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers. In my robots.txt, I have ...
2
votes
2answers
132 views

Robots.txt syntax -> Can this be done?

When creating a robots.txt, are the following the same? # Slurp User-agent: Slurp Disallow: / and # Slurp User-agent: Slurp Disallow: / It seems like for someone disallowing a large number of ...
1
vote
2answers
98 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
2
votes
2answers
319 views

Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?

Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
1
vote
2answers
379 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
0
votes
2answers
135 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
2
votes
2answers
133 views

Pointing to Google that a specific website is for mobiles

I have 2 websites: let's call them A and B. A is a Desktop optimized website. B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A How ...
1
vote
1answer
744 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...