0

I have the following robots.txt file:

User-agent: * Sitemap: https://m.mysite.com.au/sitemap.xml

enter image description here

and when I use the Webmaster Tools to validate it I get it showing that the user agent line is incorrect.

However when I validate it, it comes back as "Allowed".

In both cases it is the same file. So my questions are:

  1. What is wrong with the User-Agent line?
  2. I am being asked to remove the User-Agent line. While I understand that * means all agents I don't understand why it has to be removed.
  3. Any ideas why Google is giving inconsistent results?
3

This is because...

User-agent: *
Sitemap: https://m.mysite.com.au/sitemap.xml

...would be incorrect.

The correct syntax for the robots.txt is:

To disallow all:

User-agent: *
Disallow: /

To allow all:

User-agent: *
Disallow:

To disallow specific directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

Examples taken from here: http://www.robotstxt.org/robotstxt.html

The correct syntax for the site map is:

Sitemap: https://m.mysite.com.au/sitemap.xml

You cannot restrict sitemaps using the robots.txt file. You should place the sitemap last though I am not sure that is a requirement.

If you wish to allow all and specify a sitemap, it would be:

User-agent: *
Disallow:

Sitemap: https://m.mysite.com.au/sitemap.xml

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.