5

By now I am fighting through the world of SEO and run into a problem, I think.

Let's say I have a page like www.example.com/i-am-a-pretty-url, which is re-written by .htaccess from i-am-ugly-url.php to the above version.

If I now want google to NOT index the pretty url, will it take an effect, if I add <meta name="robots" content="noindex" /> to the i-am-ugly-url.php file?

I think it should, because there is no other way to prevent the indexing of a re-written URL, right?

However I tried this method on some "not-to-index" URLs of mine to remove them from Google index and redirected them to a remove.php, which has the meta-tag no-index in it. I fetched the URLs as Google to crawl these pages quickly multiple times, but there was no effect for days.

If the crawler not consider the meta tag, then every rewritten URL will be indexed, if the crawler finds the page anywhere. If it does and I remove the no-index-tag from the resource file, both URLs - the pretty URL and the file (*.php) will be indexed, if mentioned anywhere.

So how I solve this dilemma?

UPDATE:

.htaccess sample:
RewriteEngine on
...
...
RewriteRule ^(.*)blog/$ $1artikel.php
...
...

thats what I want:
www.example.com/blog/ -> indexed
www.example.com/article.php -> not indexed

Improve this question
11
  • 1
    This may be your problem and redirected them to a remove.php As well, it takes time for Google to remove a page from the index. Google does want to remove pages as quickly, however, the web is a huge place!!
    – closetnoc
    Apr 29, 2016 at 0:36
  • @closetnoc So you think it is only a matter of time? In that case, if I no-index my resource-file, does it also affect my pretty URL? If yes, is there a way I can manage it, that my resource file won't be indexed and my pretty-url will?
    – BeAnonymous
    Apr 29, 2016 at 0:46
  • Also, you are redirecting the request to another page. It is quite possible that you are telling Google to deindex remove.php and not your pretty URL.
    – closetnoc
    Apr 29, 2016 at 0:52
  • And how I can solve this problem? If I want to deindex the pretty URL.
    – BeAnonymous
    Apr 29, 2016 at 0:55
  • There is a difference between rewrite and redirect. You can rewrite a request and that should work, however, if you use a redirect, a redirect (either 301 or 302) is sent back. Can you post a sample of your .htaccess code?? I will let one of your experts answer. I give 301 redirects or 404s only and do use noindex at all.
    – closetnoc
    Apr 29, 2016 at 0:59

1 Answer 1

Reset to default
1

There is a way to set headers when rewriting. This is based on this Stackoverflow response and I used it successfully for one of my customer.

An example of implementation would be to set an environement variable while rewriting :

RewriteRule PATTERN DESTINATION [ENV=NOINDEX:true]

And then treat all requests tagged with this environment variable (NOINDEX) to add the X-Robots-Tag header:

Header set X-Robots-Tag "noindex, follow" env=NOINDEX

You can read more information about this header that replaces the meta-tag on this Google page, but be aware that few search engines might not support it yet.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.