0

I want to prevent pagination URLs from being crawled or indexed anywhere.

When I check site:example.com, the search results looks like this.

  • /blog/page/69/
  • /blog/page/55/

I have not listed this type of URLs in sitemap.xml

How do I stop this from happening?

  • Google doesn't use sitemaps to decide what to index. Google probably won't index everything in your sitemap. If they find other pages on your site with their crawler, they will probably decide to index some of those. Sitemaps just don't work like you are expecting them to. In fact, they have very little use for SEO at all. See The Sitemap Paradox – Stephen Ostermiller Nov 20 '19 at 15:38
2

You should firstly decide, what to do at first: prevent indexing, or prevent crawling? Both at the same time isn't possible.

Lets assume, your pagination is already indexed. Than you should firstly de-index it, and then prevent crawling. Do it on the following way:

  1. De-index.

1.1. If you can, add to each paginated page the meta tag <meta name="robots" content="noindex".

1.2. Otherwise add to your htaccess (in case of Apache 2.2) or server configuration file (in case on Nginx) a rule like:

SetEnvIf Request_URI "/page/.*" NOINDEXFOLLOW
Header set X-Robots-Tag "noindex, follow" env=REDIRECT_NOINDEXFOLLOW

1.3. For Apache 2.4 use

<If "%{REQUEST_URI} =~ m#/page/.*#">
Header set X-Robots-Tag "noindex, follow"
</If>

1.4. Note, if use Wordpress, set this rule on the beginning of htaccess file.

1.5. Create an additional sitemap with paginated page and submit it to GSC - on this way Google understands faster, what to do.

1.6. Than monitor SERP. After you realize, paginated pages aren't no longer in SERP, than:

  1. Remove any deindexing rules,

  2. Add to your robots.txt file a line Disallow: /page/*

  • 1
    the 1 2 3 order is important because if you Disallow first, robots wont be able to crawl pages and find the information you want to deindex them so they won't 🤔 – gael Nov 20 '19 at 15:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.