I have a web site where some internal links take you to another pages of the same site, passing some parameters.

<a href="/BookMgm/PreparePayment?quantity=1&itemId=123">Book</a>

My issue is I don't want crawlers follows these links.

I know I must not use the "nofollows" rel for internal links, so I tried with robots.txt:

User-agent: * 
Disallow: /backoffice/
Disallow: /bookmgm/

Unfortunatelly I continue to receive google search console errors like:

Server errors:
  • 1
    URLs are case sensitive. Does your URL actually have capital letters in it? Because your robots.txt does not. – Stephen Ostermiller Oct 3 '19 at 14:44
  • Even if you prevent Googlebot from crawling these URLs in the future, you may see reports of server errors from the URLs it has already crawled for months. – Stephen Ostermiller Oct 3 '19 at 14:45
  • @StephenOstermiller, I was not aware robots.txt is case-sensitive. I just fixed it. Let's see what happen. – Gianpiero Caretti Oct 3 '19 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.