Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
18
votes
2answers
5k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
10
votes
1answer
1k views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
8
votes
4answers
18k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
9
votes
4answers
540 views

Robots denied by domain is still listed in search results

So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results?, or any other similar question). However, if ...
6
votes
3answers
16k views

How to resolve Google “Indexed, though blocked by robots.txt”

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
11
votes
4answers
13k views

Hide a Subdomain from Google/Search Engine SEO Result?

I have a subomain which I do not want to be listed in any search engine result. Let say I have: http://www.example.com http://child.example.com How can I hide all URLs of child.example.com domain ...
5
votes
6answers
289 views

How do you prevent web pages from appearing in the Google search index?

I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a noindex tag? If so, then how do I do so?
23
votes
1answer
15k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
15
votes
4answers
6k views

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing? http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q My website was hacked which is now ...
12
votes
3answers
6k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
17
votes
7answers
8k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented Googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
12
votes
3answers
533 views

Robots.txt: do I need to disallow a page which is not linked anywhere?

There are some pages on my website that I want the user to be able to visit only if I give him/her the URL. If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
9
votes
1answer
4k views

Should I block Wordpress archive pages from search engines?

I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives. Google has indexed the site fully. Since the archives currently display full posts I've thought that I shouldn't ...
6
votes
1answer
249 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
4
votes
2answers
2k views

What is the correct way to write my “robots.txt” file?

I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: Sitemap: http://example.com/sitemap.xml Is my ...
4
votes
2answers
671 views

Should I prevent search engines indexing tag/category pages?

On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
1
vote
1answer
216 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion). I am building other sites, ...
5
votes
4answers
3k views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it is ...
5
votes
3answers
108 views

SEO Removal Time?

I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here. I believe it will be the solution but what i still need to ...
5
votes
1answer
97 views

Do we need to block repeated pages content for SEO relevance

I have multiple purchase pages with the same content like: product1red.php product2green.php Should I block them with robots.txt ?
4
votes
1answer
596 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
3
votes
2answers
362 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
2
votes
2answers
262 views

robots meta not blocking indexing

We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting ...
2
votes
2answers
1k views

Google is indexing pages with a “noindex” robots meta tag

I have come across a very annoying problem. I have several pages that shouldn't be indexed, as they are basically pop-up pages and "thanks for signing up" pages. I have set them up with the robots ...
1
vote
1answer
364 views

CDN virtual subdomain causes duplicated content

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act ...
1
vote
1answer
5k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
29
votes
6answers
501 views

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? Or can the lack of one be harmful?
10
votes
5answers
14k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
28
votes
2answers
7k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
9
votes
10answers
24k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
7
votes
4answers
9k views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
14
votes
3answers
3k views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...
11
votes
5answers
16k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
9
votes
3answers
17k views

How can I use robots.txt to disallow subdomain only?

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and ...
8
votes
2answers
11k views

What's the proper way to handle Allow and Disallow in robots.txt?

I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
6
votes
1answer
253 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
4
votes
2answers
557 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
7
votes
4answers
2k views

robots.txt not updated

I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
7
votes
5answers
1k views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
7
votes
3answers
4k views

Robots.txt for multiple domains with same website

I have the following three domains: example1.com (Disallow) example2.com (Disallow) example3.com (allow) All domain points to same folder which is public_html. How to disallow search engines ...
6
votes
2answers
4k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
6
votes
1answer
792 views

How to tell bots to forget a website and reindex it from scratch

I was wondering if there was anything I could add to robots.txt or sitemap to tell bots to completely forget everything they know about a website and index it from scratch? Context: After replacing a ...
6
votes
1answer
4k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
5
votes
3answers
1k views

Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources

I have a robots.txt file: User-agent:* Disallow:/path/page Disallow:/path/ Allow:/ The disallowed path is still getting crawled. I have searched this problem and what they said, order of precedence ...
5
votes
2answers
304 views

Preventing search engines to crawl on specific folders

I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow: User-...
4
votes
2answers
475 views

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google? Yesterday I ...
4
votes
1answer
1k views

How to have 2 sitemaps - 1 for main site the other for wordpress in subfolder

Background My site runs on a dedicated server and and has Wordpress in a subfolder. I have my mainsite which is not wordpress and that has a sitemap - for example example.com/sitemap.xml ... that ...
4
votes
2answers
15k views

Why does my sitemap have a “.gz” extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
3
votes
2answers
464 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
3
votes
3answers
657 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...