When i have just tested my latest sitemap using the google websmaster sitemap tester, i am getting a warning saying it contains URLs which are blocked by my robots.txt file, however when i check my sitemap for the example it has given me it doesn't contain them urls, so how can i fix this issue as it says there are over 600 problems.

This is one of the url which Google says returns a 404 error /catalog/product/view/id/19470

It returns a 404 error as this isnt a page any more and also all /catalog/product get blocked by my robot.txt file, however it isnt in my sitemap so i dont know where it is finding this link?

  • i have just tested my latest sitemap How?? You need to share some information such as your robots.txt file and some sample sitemap entries that fail. Otherwise we cannot even begin to guess. Anything offered could be seriously flawed. – closetnoc Nov 25 '15 at 16:10
  • i have updated my question with a bit more detail but you shouldnt need my robot.xml as im just asking why is google looking for this page when it is not even on the site map – Adam Allen Nov 25 '15 at 16:18
  • AFAIK site maps are not LIMITING, they are starting points. I.e. If you have a site map, then google will check all those listings -- eventually. But something that is NOT on the sitemap but has a link on one of your pages WILL be looked for. – Sherwood Botsford Dec 2 '18 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.