New answers tagged

0

how many requests bots make in a given time frame This time is vary for each site and each file type, based on many factors. Google calculates the average server response time for each file type and crawls them dependently of response time. Assuming your HTML files are flying from server, but your images have longer answer time, because they should be ...


11

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


4

Your robots.txt should be in the root directory and should not have any other name. According to the standard specification: This file must be accessible via HTTP on the local URL "/robots.txt".


0

My approach involves two passes: Filter only web browsers and consoles by matching the start of the user agent string with Mozilla|Opera|PSP|Bunjalloo|wii. Thanks to the user agent spoofing this check will detect almost all browsers Exclude bots by common stop strings bot|crawl|slurp|spider So if the first step is passed we assume that it is a browser ...


1

Find out how many videos viewed by normal users per hour. Let's assume It is 10 videos viewed by most of visitors per hour. Store & use the # of videos viewed using session + cookie + IP address. If the number of videos viewed are higher than 10 for any session, then use simple captcha to avoid the bots. If the captcha verification is failed ( ...


2

A few more methods of bot traffic detecting: Verification of user agent (it is a client application that uses a particular network protocol). Looking for a highly specific match to something like a malware signature or specific executable or C&C connection address. Examining such behavioral parameters as a depth of view, the duration of visit, ...



Top 50 recent answers are included