Tag Info

New answers tagged

1

Enable logging of your visitors in your site , and check whether they are coming from different ipaddress or same. If they are not from same IP then most probably they are genuine traffic


1

If it is natural, then no issues. But Bounce rate should not exceed 40% and above.


3

Bots will be sad when they only find a 404. Humans will be sad when they don’t find one. Technologies will be sad when they don’t get praised. Contributors will be sad when they are credited nowhere. Services will be sad when they can’t list or list or … your site. Webmasters will be sad when sad bots create so much overhead.


-1

No, this is completely pointless and confuses the reasons for robots.txt. The whole website by default is already for humans. Info about who made the website already has a place in the website itself either visibly or in the meta tags. Using an unconventional magic file is not good practice.


0

No difference at all as far as crawlers are able to access them (with all your valid rules) with HTTP 200 response.


0

Based on your suggestions I made some changes. Let me note that in addition that we would like not to expose areas which we do not want to crawl (by disallow / and then just specifying what we allow), we have big issue that we have over 500 bots which come in and destroy our caching system. We do not care for any bots except for google, bing and yahoo. ...


2

There is no difference to search engines between those two methods. You can generate it like you want. Moreover, you need to know blocking a page with robots.txt is not the perfect way to prevent indexing of pages. You can read this question for more information.


3

First thing: If the knowledge of "private" URLs "can be used against" you, robots.txt is the wrong tool for the job. It would be safe to assume that only a small part of all bots honour your robots.txt rules. Second thing: Note that Allow is not part of the original robots.txt specification. Some bots support it, others not. Those that don’t support Allow ...


0

Wow. There is a lot here. I appreciate your ingenuity but I suspect that you may be putting too much energy into what should be a rather simple effort. Your first example, of the two is best though either should work the same. However, I question the wisdom of blocking the home page of your site. Without knowing more about your site configuration, I am at a ...


5

Well, the first thing you should realize is that robots.txt is a standard, not a security protocol. Anything on your site that isn't secured can be crawled by a crawler/robot. The only thing robots.txt will do is tell well-behaved crawlers (e.g. GoogleBot) what you would like them to ignore. Second, I'll recommend running any robots.txt you come up with ...


6

The robots.txt spec says prefix matching is used. This means if you don't want the excluded URLs to be visible in robots.txt, you can simply abbreviate them just enough to not match any allowed URLs. For example, if you want to disallow /very/secret, you could simply use: Disallow: /ve In your robots.txt file, and as long as you don't have other URLs ...


0

It would certainly hurt your SEO for code search engines.


1

You don't state the reason to do that, so the general answer is no. JavaScript files are not usually crawled, search engines know they are there, but they usually don't do anything with them because they don't have any content for users. So there is no need to do that from the Search Engines/Crawlers perspective. From the perspective of normal users, it ...


2

There is a way to directly block the viewing of the images using .htaccess RewriteEngine on RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost [NC] RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost.*$ [NC] RewriteRule \.(gif|jpg)$ - [F] This is going to return a 403 Forbidden error if you access the image directly, but it does allow them to be ...


1

These are not Googlebot IP addresses. It is not uncommon for a bad bot operator to use Googlebot as an agent name to make you think they are okay. I found that a lot of bad bot operators come from Amazon IP addresses though I would not considering blocking Amazon IP addresses except one at a time. The Apache documentation can be found here: ...


1

Disallow: /article/ will block the URL (or page) article/ and all subfolders or subpages. That's why, if the robots.txt of externaldomain.com applies this directive for the user agent is googlebot, don't worry. Here are Google instructions on this.


8

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt



Top 50 recent answers are included