1

All of my webpages are showing ?mode=grid & ?mode=list in the google coverage. But submitted sitemap shows normal URLs. For example:

  1. example.com/page/?mode=grid
  2. example.com/page/?mode=list
  3. example.com/page/ ---> [url in sitemap]

And the robots.txt has a command Disallow: /*? which has led to blocking of all webpages from index. I don't want to remove the disallow command. How can I get the webpages indexed, removing command will show ?mode=grid & ?mode=list in google searches. Also this is a WordPress website.

  • 1
    Disallowing in robots.txt doesn't block indexing, it only blocks crawling. Google usually won't index URLs it can't crawl, but sometimes it does – Stephen Ostermiller Jun 20 at 14:43
  • Also, if Google can't crawl, it can't see your canonical tag. So it does not know that it is the wrong URL to index. – Tony McCreath Jul 9 at 9:45
0

Thinking outside of the box you may be able to apply regex to your robots.txt file to disallow any variable parameters such as https://example.php/?p=x. You may be able to filter the question mark indicating a variable in your robots.txt with regex in the file itself. I actually just Googled this and although the robots.txt file does not allow regex it looks like Google crawlers allow pattern matching. This may be the answer you are looking for. See https://searchengineland.com/a-deeper-look-at-robotstxt-17573

| improve this answer | |
  • Why do you suggest regex before saying that robots.txt doesn't support them? – Stephen Ostermiller Jul 8 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.