My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
Tl:Dr Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin. The Problem It seems at least once a week we're getting a big ...
The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
Background Our website is being a. scraped - complete pages downloaded b. spidered - complete pages and sub links followed c. mined - specific files downloaded at high speed by hundreds of ...
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...