5

By now I am fighting through the world of SEO and run into a problem, I think.

Let's say I have a page like www.example.com/i-am-a-pretty-url, which is re-written by .htaccess from i-am-ugly-url.php to the above version.

If I now want google to NOT index the pretty url, will it take an effect, if I add <meta name="robots" content="noindex" /> to the i-am-ugly-url.php file?

I think it should, because there is no other way to prevent the indexing of a re-written URL, right?

However I tried this method on some "not-to-index" URLs of mine to remove them from Google index and redirected them to a remove.php, which has the meta-tag no-index in it. I fetched the URLs as Google to crawl these pages quickly multiple times, but there was no effect for days.

If the crawler not consider the meta tag, then every rewritten URL will be indexed, if the crawler finds the page anywhere. If it does and I remove the no-index-tag from the resource file, both URLs - the pretty URL and the file (*.php) will be indexed, if mentioned anywhere.

So how I solve this dilemma?

UPDATE:

.htaccess sample:
RewriteEngine on
...
...
RewriteRule ^(.*)blog/$ $1artikel.php
...
...

thats what I want:
www.example.com/blog/ -> indexed
www.example.com/article.php -> not indexed

  • 1
    This may be your problem and redirected them to a remove.php As well, it takes time for Google to remove a page from the index. Google does want to remove pages as quickly, however, the web is a huge place!! – closetnoc Apr 29 '16 at 0:36
  • @closetnoc So you think it is only a matter of time? In that case, if I no-index my resource-file, does it also affect my pretty URL? If yes, is there a way I can manage it, that my resource file won't be indexed and my pretty-url will? – BeAnonymous Apr 29 '16 at 0:46
  • Also, you are redirecting the request to another page. It is quite possible that you are telling Google to deindex remove.php and not your pretty URL. – closetnoc Apr 29 '16 at 0:52
  • And how I can solve this problem? If I want to deindex the pretty URL. – BeAnonymous Apr 29 '16 at 0:55
  • There is a difference between rewrite and redirect. You can rewrite a request and that should work, however, if you use a redirect, a redirect (either 301 or 302) is sent back. Can you post a sample of your .htaccess code?? I will let one of your experts answer. I give 301 redirects or 404s only and do use noindex at all. – closetnoc Apr 29 '16 at 0:59
1

There is a way to set headers when rewriting. This is based on this Stackoverflow response and I used it successfully for one of my customer.

An example of implementation would be to set an environement variable while rewriting :

RewriteRule PATTERN DESTINATION [ENV=NOINDEX:true]

And then treat all requests tagged with this environment variable (NOINDEX) to add the X-Robots-Tag header:

Header set X-Robots-Tag "noindex, follow" env=NOINDEX

You can read more information about this header that replaces the meta-tag on this Google page, but be aware that few search engines might not support it yet.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.