New answers tagged

1

Use your chosen language to detect the user agent. Most legitimate search engine crawlers have their own user agent to identify it as a crawler. Then you can simply ignore those user agents.



Top 50 recent answers are included