1

In order to redirect my www.site.com to non-www, I employed this bit of code found somewhere on the web.

RewriteCond %{HTTP_HOST} ^www.website.com [NC]
RewriteRule ^(.*)$ http://website.com/$1 [L,R=301]

Although the site redirects fine, I get the "syntax not understood" warning in Google Webmaster Tools under:

webmaster tools-> Crawl-> Blocked URLs->robots.txt analysis

Google Webmaster Tools detects the sitemap just fine. I guess not much is wrong because of this, but is there a way to correct this, as I am getting errors like below:

Line 20: RewriteCond %{HTTP_HOST} ^www.website.com [NC]     Syntax not understood
Line 21: RewriteRule ^(.*)$ http://website.com/$1 [L,R=301] Syntax not understood

This is how it displays when I visit the robots.txt directly.

User-agent: *

Disallow: /cgi-bin/
Disallow: /jwp/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?

# REDIRECTING WWW. TO NON-WWW
RewriteCond %{HTTP_HOST} ^www.website.com [NC]
RewriteRule ^(.*)$ http://website.com/$1 [L,R=301]

Sitemap: http://website.com/sitemap_index.xml
  • what happens when you attempt to read website.com/robots.txt in your browser? – Wayne Jan 12 '14 at 5:59
  • @Wayne I edited my question in order to add more context to the issue and answer your question as well. – gurung Jan 12 '14 at 7:17
2

this belongs in .htaccess not robots.txt

# REDIRECTING WWW. TO NON-WWW
RewriteCond %{HTTP_HOST} ^www.website.com [NC]
RewriteRule ^(.*)$ http://website.com/$1 [L,R=301]

and "website" needs to be your website not website.com.

If when you type www.yourwebsite.com it redirects to yourwebsite.com then .htaccess is fine. Just take it out of robots.txt. If the site is not redirecting check your cpanel first to see if the redirect control exists there.

This line should be removed.

Disallow: /*?

unless you want to block the entire site!.

Are you sure you want to block category pages?

Disallow: /category/*/*

If not remove that line.

  • wow..that cleared a lot of things. Thanks for your time. – gurung Jan 12 '14 at 8:03
  • Disallow: /*? looks like a wildcard rule that will block any URL containing a query string (containing a question mark). I don't see how it would block the entire site. – Stephen Ostermiller Jan 12 '14 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.