I built a dynamic sitemap generator on my platform. By mistake I generated some 300+ wrong url along with the right ones. And they have been there for couple weeks now. Recently I found out this mistake. Some of those wrong urls are already indexed by google. I spoke with one SEO specialist and he told me to follow two steps:

  1. Add all of those wrong url to Remove Url on web masters.
  2. Next Disallow all those url from robots.txt.

I did the first step. I don't know how to do the second step. I have some 300+ urls like bellow:


Currently I cant implement 404 on these urls from coding side. Its quiet complicated now. So could any body please tell me how can I disallow these from robots.txt ??

  • 2
    Do you need to allow all category_ids or are there some that still need to be crawled and indexed? Commented May 12, 2017 at 15:09
  • 1
    There are some that needs to be crawled Commented May 12, 2017 at 15:09

1 Answer 1


You have to define first the engines you want them to skip your pages with user-agent (use * for all engines). Then you can specify the list of relative urls you want to remove. Each element preceded by Disallow:

User-agent: *
Disallow: /equip?category_id=semi-automatic
Disallow: /equip?category_id=automatic
Disallow: /equip?category_id=other

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.