Tag Info

Hot answers tagged

22

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


3

It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to. However it is possible your site will still be displayed in search results, ...


3

Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt


2

You should be doing a 301 redirect from the .php URLs to the rewritten URLs. If that is not possible for some reason you need to add a canonical URL to the .php URL pointing to the rewritten URL.


2

So there are already some answers given. For the second set, the one with the http://url in the request, these tries to find badly configured (not secured) proxy servers that could be wide open. These can then be used to hide the real origin of attacks/scans directed onto an other machine. The other machine will then see the attack as if it was coming from ...


1

If your site already had 10 million pages indexed, I would say double-check that your site is crawlable and a sitemap is submitted through Google Webmaster Tools. Google does prefer sitemaps for larger sites such as yours. It takes time for Google to notice changes sometimes especially if there is a major change with a ton of pages. It can take as much as ...


1

You could try and promote the site a bit more and get more inbound links, and/or ensure that your server can cope with a bigger crawl. See below, (but the whole answer linked is worth a read) : There is also not a hard limit on our crawl. The best way to think about it is that the number of pages that we crawl is roughly proportional to your ...


1

Okay. Most of these access (at least) are landscaping attempts to find potential vulnerabilities on your server. They are trying to fingerprint your server to know what web-based applications are installed. The first set and third set are clearly landscaping. The second set may be a result of forged request headers but still likely to be landscaping- not ...


1

The data from server logs is limited, and will unavoidably have a high noise to signal ratio, thanks to factors such as bots, caching, CDN. Analyzing page views is a task for page-tag based analytics.


1

If you're using Apache web-server you could use an .htaccess configuration to white-list by User Agent and prevent genuine bots from reaching your 'tarpit': RewriteEngine On RewriteCond %{HTTP_USER_AGENT} SEMrush [NC] RewriteRule .* - [F,L]



Only top voted, non community-wiki answers of a minimum length are eligible