Website is redirected to remove the www so www.example.com is redirecting to example.com.
Question is do I need to include the www in the sitemap for the robots.txt?
-
Not what your asking, but FYI adding sitemap to robots.txt in general is pretty useless, since you tell Google and Bing where the sitemap is in Webmaster Tools. – Simon Hayter♦ May 12 '16 at 14:01
-
Do I need to include the www? Short answer. No. – closetnoc May 12 '16 at 14:46
-
@SimonHayter: … and all other search engines (current ones and those to come in the future) are out of luck and can’t find the sitemap? I’d say it’s the best practice to reference the sitemap in the robots.txt. – unor May 12 '16 at 20:23
-
The other search engines such as DuckDuckGO have no problems crawling the site, those that can't are not worth considering anyhow. – Simon Hayter♦ May 12 '16 at 21:31
-
Found several articles claiming it to be best practice to include a sitemap reference in the robots.txt and submit via console for faster indexing. – CZorio May 13 '16 at 13:06
In your situation, Don't include "www" because web crawlers that do read the sitemap directive from your robots.txt will attempt to access the "www" version of your sitemap and then be redirected to the non-www version of the URL.
Not only will you be wasting a couple extra bytes by including "www", but the crawlers that parse the sitemap directive properly will be making two requests to your server: one for the www version, and one for the non-www version.
answered May 12 '16 at 16:02
-
Of course, if you were me, you would put in several strange redirects and www along with random versions of something like www1 in the sitemap just for fun. – closetnoc May 12 '16 at 17:22
