1

I built a dynamic sitemap generator on my platform. By mistake I generated some 300+ wrong url along with the right ones. And they have been there for couple weeks now. Recently I found out this mistake. Some of those wrong urls are already indexed by google. I spoke with one SEO specialist and he told me to follow two steps:

  1. Add all of those wrong url to Remove Url on web masters.
  2. Next Disallow all those url from robots.txt.

I did the first step. I don't know how to do the second step. I have some 300+ urls like bellow:

https://example.com/equip?category_id=semi-automatic
https://example.com/equip?category_id=automatic
https://example.com/equip?category_id=other 

Currently I cant implement 404 on these urls from coding side. Its quiet complicated now. So could any body please tell me how can I disallow these from robots.txt ??

  • 2
    Do you need to allow all category_ids or are there some that still need to be crawled and indexed? – Stephen Ostermiller May 12 '17 at 15:09
  • 1
    There are some that needs to be crawled – user3576036 May 12 '17 at 15:09
1

You have to define first the engines you want them to skip your pages with user-agent (use * for all engines). Then you can specify the list of relative urls you want to remove. Each element preceded by Disallow:

User-agent: *
Disallow: /equip?category_id=semi-automatic
Disallow: /equip?category_id=automatic
Disallow: /equip?category_id=other
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.