Skip to main content

Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
25 votes
2 answers
7k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
Ilmari Karonen's user avatar
9 votes
3 answers
21k views

How to resolve Google "Indexed, though blocked by robots.txt"

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
Eaten by a Grue's user avatar
10 votes
1 answer
2k views

How does "Noindex:" in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
Stephen Ostermiller's user avatar
11 votes
4 answers
23k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
user avatar
9 votes
4 answers
591 views

Robots denied by domain is still listed in search results

So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results?, or any other similar question). However, if ...
Kevin Peno's user avatar
16 votes
4 answers
7k views

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing? http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q My website was hacked which is now ...
user avatar
11 votes
4 answers
18k views

Hide a Subdomain from Google/Search Engine SEO Result?

I have a subomain which I do not want to be listed in any search engine result. Let say I have: http://www.example.com http://child.example.com How can I hide all URLs of child.example.com domain ...
夏期劇場's user avatar
5 votes
6 answers
808 views

How do you prevent web pages from appearing in the Google search index?

I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a noindex tag? If so, then how do I do so?
user avatar
4 votes
2 answers
713 views

Should I prevent search engines indexing tag/category pages?

On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
Macha's user avatar
  • 1,320
28 votes
2 answers
27k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
Sudheera Njs's user avatar
17 votes
7 answers
10k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented Googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
DDM's user avatar
  • 341
15 votes
3 answers
42k views

How can I use robots.txt to disallow subdomain only?

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and ...
alexus's user avatar
  • 292
14 votes
3 answers
7k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
Manju George's user avatar
12 votes
3 answers
610 views

Robots.txt: do I need to disallow a page which is not linked anywhere?

There are some pages on my website that I want the user to be able to visit only if I give him/her the URL. If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
martjno's user avatar
  • 223
9 votes
1 answer
6k views

Should I block Wordpress archive pages from search engines?

I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives. Google has indexed the site fully. Since the archives currently display full posts I've thought that I shouldn't ...
cherouvim's user avatar
  • 245
8 votes
3 answers
6k views

Robots.txt for multiple domains with same website

I have the following three domains: example1.com (Disallow) example2.com (Disallow) example3.com (allow) All domain points to same folder which is public_html. How to disallow search engines ...
Saurabh Bajpai's user avatar
8 votes
3 answers
12k views

What's the proper way to handle Allow and Disallow in robots.txt?

I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
Jim Mischel's user avatar
7 votes
3 answers
7k views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it is ...
Amit Rajawat's user avatar
5 votes
1 answer
104 views

Do we need to block repeated pages content for SEO relevance

I have multiple purchase pages with the same content like: product1red.php product2green.php Should I block them with robots.txt ?
user32057's user avatar
5 votes
3 answers
123 views

SEO Removal Time?

I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here. I believe it will be the solution but what i still need to ...
夏期劇場's user avatar
5 votes
2 answers
2k views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question is ...
CP3O's user avatar
  • 689
5 votes
2 answers
674 views

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google? Yesterday I ...
Mark's user avatar
  • 153
5 votes
2 answers
2k views

What is the correct way to write my "robots.txt" file?

I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: Sitemap: http://example.com/sitemap.xml Is my ...
ashutosh's user avatar
  • 316
5 votes
1 answer
263 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
BrunoLM's user avatar
  • 355
4 votes
1 answer
667 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
the's user avatar
  • 1,580
4 votes
2 answers
415 views

Impact of blocking site search from googlebot, but listing content pages in sitemap.xml

We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
adam's user avatar
  • 143
3 votes
2 answers
423 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
Jonathan Meyer's user avatar
2 votes
2 answers
289 views

robots meta not blocking indexing

We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting ...
Cleo Poulin's user avatar
1 vote
1 answer
989 views

Google Webmaster Tools is showing incorrect warnings - blocked by robots.txt

I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ? ...
sam's user avatar
  • 4,627
1 vote
1 answer
788 views

Internal Search Results: NOINDEX or robots.txt Blocking? [duplicate]

We have our own internal search results pages for our download section. These results are currently crawlable but with NOINDEX meta tags so they don't appear in the search index. People link to ...
Ted Wilmont's user avatar
1 vote
1 answer
7k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
user avatar
1 vote
1 answer
416 views

CDN virtual subdomain causes duplicated content

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act ...
user3474818's user avatar
1 vote
1 answer
228 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion). I am building other sites, ...
therobyouknow's user avatar
34 votes
3 answers
10k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
Foo Bar's user avatar
  • 453
31 votes
6 answers
533 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
Dan Dumitru's user avatar
22 votes
3 answers
10k views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...
bessarabov's user avatar
14 votes
6 answers
24k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
the's user avatar
  • 1,580
12 votes
11 answers
30k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
Sammy's user avatar
  • 223
11 votes
5 answers
16k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
Oliver Salzburg's user avatar
9 votes
1 answer
7k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
Yin Yang's user avatar
  • 245
8 votes
2 answers
29k views

Why does my sitemap have a ".gz" extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
LLL's user avatar
  • 191
8 votes
1 answer
4k views

Does the line-ending format of robots.txt matter?

Simple question: Should I make sure to use Unix line endings for my robots.txt, or does it not matter?
James Sulak's user avatar
7 votes
3 answers
37k views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
cnst's user avatar
  • 972
7 votes
1 answer
418 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
clickbait's user avatar
  • 382
7 votes
4 answers
10k views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
David Wilkins's user avatar
7 votes
5 answers
2k views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
sudo --do-it's user avatar
7 votes
4 answers
2k views

robots.txt not updated

I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
Haridharan's user avatar
7 votes
1 answer
3k views

Should I block the '_next' directory in a site using Next.js with robots.txt?

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx ...
checklist's user avatar
  • 171
6 votes
2 answers
6k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
Edgar Quintero's user avatar
6 votes
3 answers
14k views

How to remove old robots.txt from google as old file block the whole site [duplicate]

I have a website which still shows old robots.txt in the google webmaster tools. User-agent: * Disallow: / Which is blocking Googlebot. I have removed old file updated new robots.txt file with ...
Learning's user avatar
  • 711