Tag Info

New answers tagged

0

Unfortunaly - and this isn't documented by google - it works. I've a client site with a ajax adv that injected a asynchronously into body, and Google deindex that page.


1

You can externally redirect from the "ugly" (for want of a better term) URL to the "friendly" URL. However, you need to be careful of redirect/rewrite loops. One way of avoiding the rewrite loop is to check against THE_REQUEST - this contains the initial request header as sent from the client and is not changed as the rewrite engine rewrites the URL. This ...


1

A Disallow field that does not follow a User-agent field (ie. is not part of a group) is invalid. Bots should ignore this as being invalid. However, the rules that govern robots.txt are not a strict standard, so as with anything "invalid" in this respect, robot behaviour could be unpredicatble. The Original robots.txt standard (1994) simply states: The ...


2

Bots would simply ignore this. I'd expect them to look for specific user agents that they need to take note of. So Googlebot for example would look for User-agent: * or User-agent: Googlebot. So as those two are missing then the'd probably go ahead and crawl the entire site. We all know that adding user-agents is essential in robots.txt so there's no ...


1

There are several questions to answer here: What's the best way for me to take care of this situation and avoid wasting Google's crawl budget and also let my pages get the most of the weight from authority pages? As you say the parameters do not create millions of new URLs. So it would be okay to let Google crawl them. Your Canonicals do the rest of ...


1

Since you're allowing search engines to only one domain, your RewriteRules can be made simpler. Just use this: RewriteEngine On RewriteCond %{HTTP_HOST} ^example3\.com$ RewriteRule ^robots\.txt$ /robots-allow.txt [L] In the public_html folder you need to create two files. robots.txt and robots-allow.txt In robots.txt you need to add the following: ...


3

I work on a similar website where I manage this with PHP. In my .htaccess I have a line reading: RewriteRule ^robots.txt$ robots.php In robots.php I have the following: <?php header('Content-Type: text/plain; charset=utf-8'); if($_SERVER['HTTP_HOST']=='www.allowed-site.fr'){ echo 'User-agent: * Disallow: Host: ...


7

You need to conditionally serve a different robots.txt file based on which domain/host has been accessed. On Apache you can do this in .htaccess: RewriteEngine On RewriteCond %{HTTP_HOST} ^(example1\.com|example2\.com)$ RewriteRule ^robots\.txt$ /robots-disallow.txt [L] This is specific to your example, where it will serve a "disallow" robots.txt ...



Top 50 recent answers are included