3

All of my webpages are showing ?mode=grid & ?mode=list in the google coverage. But submitted sitemap shows normal URLs. For example:

  1. example.com/page/?mode=grid
  2. example.com/page/?mode=list
  3. example.com/page/ ---> [url in sitemap]

And the robots.txt has a command Disallow: /*? which has led to blocking of all webpages from index. I don't want to remove the disallow command. How can I get the webpages indexed, removing command will show ?mode=grid & ?mode=list in google searches. Also this is a WordPress website.

2
  • 2
    Disallowing in robots.txt doesn't block indexing, it only blocks crawling. Google usually won't index URLs it can't crawl, but sometimes it does Commented Jun 20, 2020 at 14:43
  • Also, if Google can't crawl, it can't see your canonical tag. So it does not know that it is the wrong URL to index. Commented Jul 9, 2020 at 9:45

3 Answers 3

2

The Disallow directive doesn't ask search engines not to INDEX a resource, just not to crawl it. If google finds a link to that resource, it indexes it, even without being able to scan it (enter the link graph). Also, the rel canonical tag cannot have any effect, since disallowing prevents google from reading the page (and thus finding the canonical rel). In order not to index a page you should use NOINDEX, or a rel canonical, but NEVER a Disallow.

1

I had the same issue and it turned out that if a page uses the parameters then the page with the param may get indexed. Once indexed, Google kept preferring the parametrized page over the non-parameter version.

I took multiple measures and I am getting rid of them. Use the measure that are appropriate for you -

  1. Use the Google Removal Tool to remove the param version of the url
  2. Stop internal (and external) linking to param versions
  3. Do a 301 redirect from the param version to the non-param version. (Or return a 404, it depends on your situation).
  4. Ensure the canonical in the param version points to the non param version.
0

Thinking outside of the box you may be able to apply regex to your robots.txt file to disallow any variable parameters such as https://example.php/?p=x. You may be able to filter the question mark indicating a variable in your robots.txt with regex in the file itself. I actually just Googled this and although the robots.txt file does not allow regex it looks like Google crawlers allow pattern matching. This may be the answer you are looking for. See https://searchengineland.com/a-deeper-look-at-robotstxt-17573

1
  • Why do you suggest regex before saying that robots.txt doesn't support them? Commented Jul 8, 2020 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.