Tag Info

Hot answers tagged

3

To prevent misuse the inner workings of Googlebot are kept secret so there is no real "expected behaviour". While searching the net I stumbled upon the following: ... search engines understand URL parameters but often ignore fragments. This implies URL fragments might get crawled. On another page I found a second hint. I always thought Googlebot ...


2

Robot traffic does not affect page rank. Whilst robots can & may index your website, the sheer volume of robots hitting your website does not play any role whatsoever in affecting your page rank, the driving force behind Google's organic search traffic algorithm. Brian Dean shared a very, very comprehensive list of factors which influence this which can ...


2

Yes you can. In fact, I recommend it and others too. I would not work so hard on my regular expression to include version numbers- just ^PHP.*$ You will find that some agent names are consistently up to no good even though some will argue that they can be used for good and should not be blocked. That is a negative argument. I study these things and I ...


2

PHP user agents are known as libraries. It should be no problem if you block them since legitimate crawlers use other string types, however, can't you check and block only the IP's? In this site, for example, you have ways to block only Chinese traffic.


2

Bing may change titles in their SERPs (Just like Google). See Bing’s blog post How Does Bing Choose The Title For My Web Page?: Sometimes, despite a webmaster’s best efforts, Bing may choose to serve a title that is different to the title of a web site or document. Why "My social networks"? Maybe because it is your first heading on the homepage, which ...


1

Google can crawl and index some content put into the page with the embed tag. For example Google says that they index flash files that are embedded Googlebot can index almost any text a user can see as they interact with any Flash SWF file on your site, and can use that text to generate a snippet or match query terms in Google searches. Additionally, ...


1

The way Google says to handle it makes me think they don't appreciate iframes that much. Google supports frames and iframes to the extent that it can. Frames can cause problems for search engines because they don't correspond to the conceptual model of the web. In this model, one page displays only one URL. Pages that use frames or iframes display ...


1

If your site already had 10 million pages indexed, I would say double-check that your site is crawlable and a sitemap is submitted through Google Webmaster Tools. Google does prefer sitemaps for larger sites such as yours. It takes time for Google to notice changes sometimes especially if there is a major change with a ton of pages. It can take as much as ...


1

You could try and promote the site a bit more and get more inbound links, and/or ensure that your server can cope with a bigger crawl. See below, (but the whole answer linked is worth a read) : There is also not a hard limit on our crawl. The best way to think about it is that the number of pages that we crawl is roughly proportional to your ...


1

When a user has the Alexa Toolbar installed, "Alexa" appears in the user agent string of the browser. You could deny those users access to your website using the following code in your .htaccess file: BrowserMatchNoCase Alexa alexa_user_agent Order Deny,Allow Deny from env=alexa_user_agent I wouldn't recommend doing this, it is going to inconvenience ...



Only top voted, non community-wiki answers of a minimum length are eligible