1

I have created robots.txt in my root directory and when I navigate to my website address like https://example.ca/robots.txt I am able to see the rules

User-agent:*
Disallow: /captcha/
Disallow: /mailing/
Disallow: /assets/

Allow: /

Sitemap: https://domain.ca/sitemap.xml
Sitemap: http://domain.ca/sitemap.xml

But when I want to test the file in Google robots.txt Tester

enter image description here

I am getting this error

enter image description here

I am however able to test the file at bottom of the page or see the live robots.txt

enter image description here

but still getting 404 error when entering to the page!

Improve this question
2
  • Wen you say you created the robots.txt, when did you do that? Was it just created today?
    – Stephen Ostermiller ♦
    May 11 at 7:33
  • Yes it is newly created file
    – Behseini
    May 11 at 14:21

1 Answer 1

Reset to default
3

Can take up to 24 hours, the robot tester seems to a bit flakey after just putting your file up. Good thing at least is that you're not getting a BLOCKED response from the tester.

https://developers.google.com/search/docs/advanced/robots/submit-updated-robots-txt

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.