Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

33
votes
4answers
3k views

Files I should have on every website

I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs. So far I have the ...
27
votes
6answers
421 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
21
votes
2answers
3k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
14
votes
2answers
4k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
14
votes
7answers
5k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
11
votes
3answers
4k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
11
votes
2answers
904 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
11
votes
1answer
811 views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...
10
votes
3answers
429 views

Robots.txt: do I need to disallow a page which is not linked anywhere?

There are some pages on my websites that I want the user to be able to visit only if I give him/her the url. If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
10
votes
1answer
3k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
10
votes
3answers
520 views

Does Google Preview obey Robots.txt?

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky. Is this the case and is there a way to ...
9
votes
1answer
6k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. User-...
8
votes
2answers
824 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
8
votes
4answers
517 views

Robots denied by domain is still listed in search results

So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question). However, if ...
8
votes
2answers
6k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
8
votes
5answers
3k views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
8
votes
1answer
203 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
8
votes
4answers
15k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
7
votes
7answers
3k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
7
votes
5answers
595 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
7
votes
4answers
1k views

robots.txt not updated

I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
7
votes
2answers
188 views

Is this “robots.txt” file really preventing all crawling of our website? I'm trying to find out why our SEO is so poor

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following: # go away User-...
7
votes
1answer
13k views

How do I disallow an entire directory with robots.txt?

I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example http://someurl.com/first_url http://someurl.com/...
7
votes
2answers
9k views

What's the proper way to handle Allow and Disallow in robots.txt?

I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
7
votes
1answer
2k views

Should I block Wordpress archive pages from search engines?

I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives. Google has indexed the site fully. Since the archives currently display full posts I've thought that I shouldn't ...
7
votes
4answers
398 views

How do you configure robots.txt to allow crawling of the site except for a few directories?

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders? Is there a general setup that should always be used?
6
votes
4answers
1k views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
6
votes
4answers
2k views

Should I set NOINDEX header for my JS, CSS and image files?

Are there any harms if my site send NOINDEX headers for all my static assets? For image files, I refer to those valueless, e.g. background images, button images, etc. Update: more background ...
6
votes
4answers
6k views

Hide a Subdomain from Google/Search Engine SEO Result?

I have a subomain which I do not want to be listed in any search engine result. Let say I have: http://www.example.com http://child.example.com How can I hide all URLs of child.example.com domain ...
6
votes
3answers
3k views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
6
votes
3answers
798 views

Robots.txt for multiple domains with same website

I have the following three domains: example1.com (Disallow) example2.com (Disallow) example3.com (allow) All domain points to same folder which is public_html. How to disallow search ...
6
votes
4answers
6k views

In Robots.txt, how do I disallow all pages except for the index page?

I want google to index my index page, but do not index any other pages. User-agent: * Disallow: / So far I have this. However...when I do this, Google does not index my index page. When I search ...
6
votes
1answer
521 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
6
votes
2answers
4k views

How do you disallow root in robots.txt, but allow a subdirectory?

Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?
6
votes
1answer
131 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
6
votes
2answers
442 views

80% of Website Traffic coming from Just one Town in USA - Strange?

For the last month, 80% of my total traffic was coming from just one Town in the USA. Here are the attributes: It is one of the richest towns in California in USA which defies logic since I run a ...
6
votes
2answers
322 views

What kind of redirect (301 or 302) for an email links tracker?

We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application (https://...
6
votes
3answers
333 views

Does loading images from a server that disallows robots affect SEO?

Lets say a page on example.com has all of its images coming from img.example.com: <img src="http://img.example.com/keyword.jpg" alt="keyword"/> while robots.txt on img.example.com disallows ...
5
votes
5answers
4k views

Disallow robots.txt from being accessed in a browser but still accessible by spiders?

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
5
votes
2answers
177 views

Preventing search engines to crawl on specific folders

I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow: User-...
5
votes
9answers
17k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
5
votes
1answer
215 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
5
votes
2answers
140 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
5
votes
2answers
303 views

Robots.txt and pattern matching

Adding this to my robots.txt User-agent: * Disallow: /*action=*$ How does robots not recognizing wild cards handle this?
5
votes
3answers
70 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
5
votes
1answer
559 views

Robots.txt got changed to disallow crawling by accident

Our robots.txt mistakenly got changed to: User-agent: * Disallow: / This occurred about 2 weeks ago and was changed back yesterday to allow crawling. Our listed pages in the Googles have ...
5
votes
5answers
187 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
5
votes
1answer
176 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
5
votes
3answers
98 views

SEO Removal Time?

I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here. I believe it will be the solution but what i still need to ...
5
votes
1answer
780 views

Do I reference all my *.xml.gz sitemap files in robots.txt

I have a website with over 1 million url's. As such, I have 1 sitemap.xml file which acts as an index to many sitemap0000.xml.gz, sitemap0001.xml.gz ... sitemap0030.xml.gz etc. files. My question is: ...