Hot answers tagged

11

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


4

Your robots.txt should be in the root directory and should not have any other name. According to the standard specification: This file must be accessible via HTTP on the local URL "/robots.txt".


3

If you really want to prevent a link from being indexed or followed, you can go extreme as follows: If you're using a server-side scripting language or have sufficient apache access, then modify code so that the page to not be indexed will have an HTTP 410 status code attached to it, meaning the page is gone for good. This will effectively cause previous ...


2

The easiest way to solve this is to version your JavaScript files. Then only new pages will fetch the new JavaScript.


2

A few more methods of bot traffic detecting: Verification of user agent (it is a client application that uses a particular network protocol). Looking for a highly specific match to something like a malware signature or specific executable or C&C connection address. Examining such behavioral parameters as a depth of view, the duration of visit, ...


1

Nothing in the world will make a real link undiscoverable by Google. Even if you close your http://example.com/page?download from crawling with robots.txt, de-index the download page with noindex, and markup the link with nofollow - it is enough to get one single incoming backlink to your http://example.com/page?download, and the page is crawled. Thats why ...


1

how many requests bots make in a given time frame This time is vary for each site and each file type, based on many factors. Google calculates the average server response time for each file type and crawls them dependently of response time. Assuming your HTML files are flying from server, but your images have longer answer time, because they should be ...


1

Find out how many videos viewed by normal users per hour. Let's assume It is 10 videos viewed by most of visitors per hour. Store & use the # of videos viewed using session + cookie + IP address. If the number of videos viewed are higher than 10 for any session, then use simple captcha to avoid the bots. If the captcha verification is failed ( ...



Only top voted, non community-wiki answers of a minimum length are eligible