1

Correct me if I am wrong, this means bots cannot crawl the old parts of the site?

User-agent: *
Disallow: /old/
Disallow: /old

Since it has been days that I have not received any traffic and found out about the robot.txt setup.

  • 1
    Aside: Since robots.txt is prefix matching, the first Disallow directive above (with a slash) is superfluous. – MrWhite Aug 4 '16 at 7:39
3

This does not meant "old" as in "advanced age", it means /old as in directory/URL. For example, www.example.com/old/how-to-bake-special-brownies.html.

I assume /old is not your entire site, however if it is, then that would explain things.

  • since the site went to full revamped and design so I was assuming that the new robot txt setup prevented the crawlers from accessing the old directories/URL – Tony Logarta Aug 4 '16 at 0:35
  • @TonyLogarta The code you provided looks fine. Whenever you change a site, there is a disruption in search for a while as the search engine goes through your site and re-evaluates it. Each new page has to prove itself with performance metrics too. So it takes a while. – closetnoc Aug 4 '16 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.