Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
22
votes
2answers
6k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
7
votes
3answers
20k views

How to resolve Google "Indexed, though blocked by robots.txt"

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
10
votes
1answer
2k views

How does "Noindex:" in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
11
votes
4answers
21k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
11
votes
4answers
16k views

Hide a Subdomain from Google/Search Engine SEO Result?

I have a subomain which I do not want to be listed in any search engine result. Let say I have: http://www.example.com http://child.example.com How can I hide all URLs of child.example.com domain ...
9
votes
4answers
563 views

Robots denied by domain is still listed in search results

So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results?, or any other similar question). However, if ...
11
votes
3answers
29k views

How can I use robots.txt to disallow subdomain only?

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and ...
4
votes
2answers
687 views

Should I prevent search engines indexing tag/category pages?

On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
5
votes
6answers
436 views

How do you prevent web pages from appearing in the Google search index?

I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a noindex tag? If so, then how do I do so?
25
votes
2answers
20k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
16
votes
4answers
7k views

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing? http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q My website was hacked which is now ...
14
votes
3answers
6k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
17
votes
7answers
9k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented Googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
8
votes
3answers
11k views

What's the proper way to handle Allow and Disallow in robots.txt?

I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
8
votes
3answers
5k views

Robots.txt for multiple domains with same website

I have the following three domains: example1.com (Disallow) example2.com (Disallow) example3.com (allow) All domain points to same folder which is public_html. How to disallow search engines ...
6
votes
1answer
316 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
12
votes
3answers
574 views

Robots.txt: do I need to disallow a page which is not linked anywhere?

There are some pages on my website that I want the user to be able to visit only if I give him/her the URL. If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
9
votes
1answer
5k views

Should I block Wordpress archive pages from search engines?

I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives. Google has indexed the site fully. Since the archives currently display full posts I've thought that I shouldn't ...
6
votes
1answer
254 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
5
votes
2answers
528 views

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google? Yesterday I ...
4
votes
2answers
2k views

What is the correct way to write my "robots.txt" file?

I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: Sitemap: http://example.com/sitemap.xml Is my ...
1
vote
1answer
221 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion). I am building other sites, ...
6
votes
4answers
5k views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it is ...
5
votes
1answer
98 views

Do we need to block repeated pages content for SEO relevance

I have multiple purchase pages with the same content like: product1red.php product2green.php Should I block them with robots.txt ?
5
votes
3answers
114 views

SEO Removal Time?

I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here. I believe it will be the solution but what i still need to ...
4
votes
2answers
404 views

Impact of blocking site search from googlebot, but listing content pages in sitemap.xml

We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
4
votes
1answer
634 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
3
votes
2answers
386 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
2
votes
2answers
275 views

robots meta not blocking indexing

We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting ...
1
vote
1answer
6k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
1
vote
1answer
398 views

CDN virtual subdomain causes duplicated content

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act ...
1
vote
1answer
980 views

Google Webmaster Tools is showing incorrect warnings - blocked by robots.txt

I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ? ...
30
votes
6answers
512 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
13
votes
6answers
19k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
33
votes
2answers
9k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
12
votes
11answers
27k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
7
votes
4answers
9k views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
18
votes
3answers
6k views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...
11
votes
5answers
16k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
6
votes
3answers
31k views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
5
votes
1answer
227 views

What is difference between robots.txt, sitemap, robots meta tag, robots header tag?

So I am trying to learn SEO and I am honestly confused and have following 8 questions. Do I tell a bot not to visit a certain link through X-Robots-Tag or through robot meta tag or robots.txt? Is it ...
4
votes
2answers
622 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting "host-load"

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
7
votes
1answer
6k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
7
votes
4answers
2k views

robots.txt not updated

I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
7
votes
5answers
2k views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
6
votes
2answers
5k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
6
votes
2answers
23k views

Why does my sitemap have a ".gz" extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
5
votes
2answers
348 views

Preventing search engines to crawl on specific folders

I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow: User-...
5
votes
4answers
7k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
5
votes
3answers
1k views

Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources

I have a robots.txt file: User-agent:* Disallow:/path/page Disallow:/path/ Allow:/ The disallowed path is still getting crawled. I have searched this problem and what they said, order of precedence ...