Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

In order to redirect my www.site.com to non-www, I employed this bit of code found somewhere on the web.

RewriteCond %{HTTP_HOST} ^www.website.com [NC]
RewriteRule ^(.*)$ http://website.com/$1 [L,R=301]

Although the site redirects fine, I get the "syntax not understood" warning in Google Webmaster Tools under:

webmaster tools-> Crawl-> Blocked URLs->robots.txt analysis

Google Webmaster Tools detects the sitemap just fine. I guess not much is wrong because of this, but is there a way to correct this, as I am getting errors like below:

Line 20: RewriteCond %{HTTP_HOST} ^www.website.com [NC]     Syntax not understood
Line 21: RewriteRule ^(.*)$ http://website.com/$1 [L,R=301] Syntax not understood

This is how it displays when I visit the robots.txt directly.

User-agent: *

Disallow: /cgi-bin/
Disallow: /jwp/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?

# REDIRECTING WWW. TO NON-WWW
RewriteCond %{HTTP_HOST} ^www.website.com [NC]
RewriteRule ^(.*)$ http://website.com/$1 [L,R=301]

Sitemap: http://website.com/sitemap_index.xml
share|improve this question
    
what happens when you attempt to read website.com/robots.txt in your browser? –  Wayne Jan 12 at 5:59
    
@Wayne I edited my question in order to add more context to the issue and answer your question as well. –  gurung Jan 12 at 7:17
add comment

1 Answer

up vote 2 down vote accepted

this belongs in .htaccess not robots.txt

# REDIRECTING WWW. TO NON-WWW
RewriteCond %{HTTP_HOST} ^www.website.com [NC]
RewriteRule ^(.*)$ http://website.com/$1 [L,R=301]

and "website" needs to be your website not website.com.

If when you type www.yourwebsite.com it redirects to yourwebsite.com then .htaccess is fine. Just take it out of robots.txt. If the site is not redirecting check your cpanel first to see if the redirect control exists there.

This line should be removed.

Disallow: /*?

unless you want to block the entire site!.

Are you sure you want to block category pages?

Disallow: /category/*/*

If not remove that line.

share|improve this answer
    
wow..that cleared a lot of things. Thanks for your time. –  gurung Jan 12 at 8:03
    
Disallow: /*? looks like a wildcard rule that will block any URL containing a query string (containing a question mark). I don't see how it would block the entire site. –  Stephen Ostermiller Jan 12 at 12:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.