1

When I search site:[example].com in Google for my blog, the majority of the pages that are being indexed are "pretty links" to external sites.

I created these links using the Pretty Links plugin so that I can make links to external sites look nice and clean, and also so I can track how many people click through to those sites from my blog.

But many of them were created before I understood anything about follow/no follow links. As I've learned more, I realised this isn't ideal and have gone and changed each of these links to "no follow" , but this hasn't stopped then from being indexed by Google.

How can I stop these links from being index?

  • 1
    What type of redirect do these use? Is it a 301 permanent, 302 temporary, meta refresh, or something else? Also, what is in your robots.txt file for these URLs? Can Googlebot crawl them and see that they redirect? – Stephen Ostermiller Aug 3 '17 at 11:26
  • 1
    Also keep in mind that site: search results are special. Google often includes results in there that will never appear in any normal (non-site:) search result – Stephen Ostermiller Aug 3 '17 at 11:27
  • I'm not sure the answer to any of your questions other than my robots.txt file only says: Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php – Matilda Aug 3 '17 at 11:52
  • Presumably your redirect URLs don't start with /wp-admin What is an example of one? Something like /some-other-site? You can check the type of redirect curl --head http://example.com/some-other-site on the command line or using an online tool like redirect-checker.org – Stephen Ostermiller Aug 3 '17 at 11:59
  • Here's an example of one: www.[example].com/alp2. I just used httpstatus.io and it came back with this 3 redirects - 307 > 301 > 301> 200 – Matilda Aug 3 '17 at 12:41
1

Google doesn't index links, it indexes pages.

It seems that the pages that these pretty links point to like example.org/clickout/5 don't have a proper robots setting or http status code to let Google know that you don't want these pages in the index.

Have you tried the following?

  • Add a meta robots noindex to the intermediate clickout pages? This will stop Google from indexing the intermediate clickout pages.
  • Switch to non-pretty external links that are tracked using Javascript click events (for instance using Google Analytics?)
  • Use a tool like https://httpstatus.io/ to check the http status codes that the intermediate page is returning. As Stephen noted, if these are 301 it's unlikely that these clickout pages will get indexed themselves.

[Edit] Removed suggestion to block the intermediate URLs in robots.txt, added suggestion to check http status code.

| improve this answer | |
  • Thank you for your help! For someone who is not very technically savvy, do you have a resource to point me to which will break down exactly how to do the first of your suggestions: adding the clickout URLs to your robots.txt file? – Matilda Aug 3 '17 at 10:04
  • The official resource is pretty thorough yet accessible: robotstxt.org/robotstxt.html – Theo van der Zee Aug 3 '17 at 10:07
  • 2
    I would suggest NOT disallowing them in robots.txt. If you disallow them, Google will not be able to crawl them and find out they redirect. Google doesn't usually index 301 permanent redirecting URLs, but it may include URLs in the index if it finds links to them and can't crawl them. – Stephen Ostermiller Aug 3 '17 at 11:28
  • I stand corrected and edited my original post to reflect that. – Theo van der Zee Aug 3 '17 at 11:37
0

Add

<meta name="robots" content="noindex,nofollow"/>
<meta name="googlebot" content="noindex,nofollow,noarchive,unavailable_after:[date]"/>

at the head of pages you want to exclude from Google search, where [date] should be a date in the past to tell Googlebot that the page should no longer be available at present.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.