Tag Info

New answers tagged

0

I have just successfully blocked the Chinese searchbot Baiduspider from accessing any content on my site. I made the decision to do so because of the following reasons. Reasons for deciding to block Approximately every 20th request to my server was from a baidu bot. This is unpolite behavior. Baidubot accounts for 5% of my sites bandwidth usage. I make ...


1

There is no standard. You will have to consider how robust the site is. Multiple servers and lots of bandwidth would be less taxed than a single server with less bandwidth. Also consider that some are paying for the bandwidth they use rather than having a set amount rated on a per second basis (like we are used to with DSL and cable). This would mean that ...


0

We did something like this on a one page site, where each section of the site was served up via Ajax and cachable as its own url. We did this by using the #! hashbang method and then using the escaped fragment meta tag to let google know about the section pages. When the bot hit our site with the escaped fragment querystring we only served the bot that ...



Top 50 recent answers are included