Hot answers tagged

11

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


4

Your robots.txt should be in the root directory and should not have any other name. According to the standard specification: This file must be accessible via HTTP on the local URL "/robots.txt".


2

A few more methods of bot traffic detecting: Verification of user agent (it is a client application that uses a particular network protocol). Looking for a highly specific match to something like a malware signature or specific executable or C&C connection address. Examining such behavioral parameters as a depth of view, the duration of visit, ...


1

Find out how many videos viewed by normal users per hour. Let's assume It is 10 videos viewed by most of visitors per hour. Store & use the # of videos viewed using session + cookie + IP address. If the number of videos viewed are higher than 10 for any session, then use simple captcha to avoid the bots. If the captcha verification is failed ( ...



Only top voted, non community-wiki answers of a minimum length are eligible