We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
36
votes
4answers
4k views

Files I should have on every website

I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs. So far I have the ...
29
votes
6answers
484 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
28
votes
2answers
7k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
23
votes
1answer
15k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
19
votes
2answers
39k views

How do I disallow an entire directory with robots.txt?

I have two sites I am currently working on which is a shopping cart and another shopper. Both are under the same domain name so for example http://example.com/first_url http://example.com/...
18
votes
2answers
5k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
17
votes
7answers
8k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented Googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
15
votes
4answers
6k views

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing? http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q My website was hacked which is now ...
15
votes
3answers
9k views

Can robots.txt be in a server's sub-directory?

I have a sub-directory that I would like to hide from the search engine web crawlers. One way to do this is to use a robots.txt in the root directory of the server (the standard way). However, anyone ...
14
votes
3answers
3k views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...
12
votes
3answers
5k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
12
votes
3answers
533 views

Robots.txt: do I need to disallow a page which is not linked anywhere?

There are some pages on my website that I want the user to be able to visit only if I give him/her the URL. If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
11
votes
4answers
13k views

Hide a Subdomain from Google/Search Engine SEO Result?

I have a subomain which I do not want to be listed in any search engine result. Let say I have: http://www.example.com http://child.example.com How can I hide all URLs of child.example.com domain ...
11
votes
2answers
1k views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
11
votes
2answers
10k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.example.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
11
votes
5answers
16k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
11
votes
3answers
588 views

Does Google Preview obey Robots.txt?

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky. Is this the case and is there a way to ...
10
votes
2answers
16k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. User-...
10
votes
3answers
9k views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
10
votes
1answer
1k views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
10
votes
1answer
1k views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
9
votes
5answers
13k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
9
votes
4answers
540 views

Robots denied by domain is still listed in search results

So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results?, or any other similar question). However, if ...
9
votes
10answers
24k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
9
votes
3answers
16k views

How can I use robots.txt to disallow subdomain only?

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and ...
9
votes
1answer
4k views

Should I block Wordpress archive pages from search engines?

I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives. Google has indexed the site fully. Since the archives currently display full posts I've thought that I shouldn't ...
8
votes
4answers
17k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
8
votes
4answers
3k views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
8
votes
3answers
2k views

When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?

A site I am working on moved a subdomain to another subdomain via a 301 redirect. However when checking robots.txt of the old subdomain, it has a robots.txt which disallows search engine web crawlers ...
8
votes
5answers
4k views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
8
votes
1answer
2k views

Does the line-ending format of robots.txt matter?

Simple question: Should I make sure to use Unix line endings for my robots.txt, or does it not matter?
8
votes
2answers
11k views

What's the proper way to handle Allow and Disallow in robots.txt?

I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
7
votes
7answers
4k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
7
votes
5answers
1k views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
7
votes
4answers
2k views

robots.txt not updated

I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
7
votes
4answers
9k views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
7
votes
4answers
4k views

Should I set NOINDEX header for my JS, CSS and image files?

Are there any harms if my site send NOINDEX headers for all my static assets? For image files, I refer to those valueless, e.g. background images, button images, etc. Update: more background ...
7
votes
3answers
3k views

Robots.txt for multiple domains with same website

I have the following three domains: example1.com (Disallow) example2.com (Disallow) example3.com (allow) All domain points to same folder which is public_html. How to disallow search engines ...
7
votes
3answers
6k views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
7
votes
4answers
8k views

In Robots.txt, how do I disallow all pages except for the index page?

I want google to index my index page, but do not index any other pages. User-agent: * Disallow: / So far I have this. However...when I do this, Google does not index my index page. When I search ...
7
votes
2answers
211 views

Is this “robots.txt” file really preventing all crawling of our website? I'm trying to find out why our SEO is so poor

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following: # go away User-...
7
votes
2answers
6k views

How do you disallow root in robots.txt, but allow a subdirectory?

Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?
7
votes
4answers
453 views

How do you configure robots.txt to allow crawling of the site except for a few directories?

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders? Is there a general setup that should always be used?
6
votes
5answers
7k views

Disallow robots.txt from being accessed in a browser but still accessible by spiders?

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
6
votes
5answers
20k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
6
votes
2answers
4k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
6
votes
1answer
249 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
6
votes
3answers
15k views

How to resolve Google “Indexed, though blocked by robots.txt”

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
6
votes
1answer
727 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
6
votes
1answer
251 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?