Hot answers tagged web-crawlers
No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...
Your robots.txt should be in the root directory and should not have any other name. According to the standard specification: This file must be accessible via HTTP on the local URL "/robots.txt".
A few more methods of bot traffic detecting: Verification of user agent (it is a client application that uses a particular network protocol). Looking for a highly specific match to something like a malware signature or specific executable or C&C connection address. Examining such behavioral parameters as a depth of view, the duration of visit, ...
Find out how many videos viewed by normal users per hour. Let's assume It is 10 videos viewed by most of visitors per hour. Store & use the # of videos viewed using session + cookie + IP address. If the number of videos viewed are higher than 10 for any session, then use simple captcha to avoid the bots. If the captcha verification is failed ( ...
Only top voted, non community-wiki answers of a minimum length are eligible