All of my webpages are showing ?mode=grid & ?mode=list in the google coverage. But submitted sitemap shows normal URLs. For example:

  1. example.com/page/?mode=grid
  2. example.com/page/?mode=list
  3. example.com/page/ ---> [url in sitemap]

And the robots.txt has a command Disallow: /*? which has led to blocking of all webpages from index. I don't want to remove the disallow command. How can I get the webpages indexed, removing command will show ?mode=grid & ?mode=list in google searches. Also this is a WordPress website.

  • 2
    Disallowing in robots.txt doesn't block indexing, it only blocks crawling. Google usually won't index URLs it can't crawl, but sometimes it does Jun 20, 2020 at 14:43
  • Also, if Google can't crawl, it can't see your canonical tag. So it does not know that it is the wrong URL to index. Jul 9, 2020 at 9:45

2 Answers 2


I had the same issue and it turned out that if a page uses the parameters then the page with the param may get indexed. Once indexed, Google kept preferring the parametrized page over the non-parameter version.

I took multiple measures and I am getting rid of them. Use the measure that are appropriate for you -

  1. Use the Google Removal Tool to remove the param version of the url
  2. Stop internal (and external) linking to param versions
  3. Do a 301 redirect from the param version to the non-param version. (Or return a 404, it depends on your situation).
  4. Ensure the canonical in the param version points to the non param version.

Thinking outside of the box you may be able to apply regex to your robots.txt file to disallow any variable parameters such as https://example.php/?p=x. You may be able to filter the question mark indicating a variable in your robots.txt with regex in the file itself. I actually just Googled this and although the robots.txt file does not allow regex it looks like Google crawlers allow pattern matching. This may be the answer you are looking for. See https://searchengineland.com/a-deeper-look-at-robotstxt-17573

  • Why do you suggest regex before saying that robots.txt doesn't support them? Jul 8, 2020 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.