Tag Info

Hot answers tagged


If you test the robots.txt in one of the many robot.txt validators you'll see it does what you want. For instance using the seobook robot txt validator shows that when tested for the url / these bots should not spider your website. If you really want that is another question. If a bot crawling the website is too much of a strain on the resources, then ...


I noticed a newly registered domain is being hit by non-existent URLs... It's being hit for thousands per day which is causing high server load. There are ways to mitigate the load from all sources (including google) that you believe are causing it. Cut down on the amount of code in the error pages. This means no error pages with fancy templates, ...

Only top voted, non community-wiki answers of a minimum length are eligible