1

I built a dynamic sitemap generator on my platform. By mistake I generated some 300+ wrong url along with the right ones. And they have been there for couple weeks now. Recently I found out this mistake. Some of those wrong urls are already indexed by google. I spoke with one SEO specialist and he told me to follow two steps:

  1. Add all of those wrong url to Remove Url on web masters.
  2. Next Disallow all those url from robots.txt.

I did the first step. I don't know how to do the second step. I have some 300+ urls like bellow:

https://example.com/equip?category_id=semi-automatic
https://example.com/equip?category_id=automatic
https://example.com/equip?category_id=other

Currently I cant implement 404 on these urls from coding side. Its quiet complicated now. So could any body please tell me how can I disallow these from robots.txt ??

Improve this question
2
  • 2
    Do you need to allow all category_ids or are there some that still need to be crawled and indexed?
    – Stephen Ostermiller ♦
    May 12, 2017 at 15:09
  • 1
    There are some that needs to be crawled
    – user3576036
    May 12, 2017 at 15:09

1 Answer 1

Reset to default
1

You have to define first the engines you want them to skip your pages with user-agent (use * for all engines). Then you can specify the list of relative urls you want to remove. Each element preceded by Disallow:

User-agent: *
Disallow: /equip?category_id=semi-automatic
Disallow: /equip?category_id=automatic
Disallow: /equip?category_id=other
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.