Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have installed the mybb forum in subdomain of my site. The link to my site is here.

but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt file?

User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
share|improve this question
    
When did this file go live? Google might have indexed those pages before you made that robots.txt, and then it might take a while for them to disappear. –  Ivo van der Veeken Dec 11 '13 at 13:39
    
I have added the robot file more then a month ago. –  Sidh Dec 11 '13 at 14:09
1  
Are you using seo friendly urls? if so using .php in the robots is irrelevant. Additionally, as Ivo said it can take months to remove urls if these were already indexed, if concerned about these URLS then login to webmaster tools and speed up the process by removing them using the url removal tool. Additionally using 'noindex' actually within the meta along with robots is considered better and more reliable. –  bybe Dec 11 '13 at 16:11
1  
It may be helpful if you post the URL. –  Max Dec 12 '13 at 1:09
add comment

1 Answer

The best way to diagnose problems with your robots.txt file is to use Google Webmaster Tools. They have a tool there that is specifically meant to figure out this kind of issue. Once you are in GWT, navigate to:

Crawl → Blocked URLs

Once there, it will import your robots.txt file in the upper box. You can test changes by editing there (but it won't save them back to your site, of course.) The lower box is for a list of URLs that you want to test. Choose some URLs that you want to block but which are indexed and test them.


It is possible that nothing is wrong with your robots.txt file.

If you recently added rules to your robots.txt file it may take a month or more until the pages get removed from Google's index. Google doesn't instantly de-index pages once you block those pages. Google won't de-index pages until next time it tries to crawl them. If you would like them removed sooner, you can request pages be removed from the index. In GWT, navigate to:

Google Index → Remove URLs

According to their removal requirements the pages must be blocked (for example by robots.txt) before they can be removed.


robots.txt may not prevent Google from including a page in the index altogether. In some cases, when a page has enough external links, Google may include the page in the index and use the anchor text of the inbound links to know what it is about. In such cases, you can request that the page not be indexed by allowing Googlebot to crawl the page but putting a meta robots noindex tag into the head.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.