Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

10
votes
3answers
3k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
9
votes
2answers
299 views

How is Googlebot finding URLs that are only visible to authenticated users?

Here is one of my customers, performing some action after having logged in to his account. The unique token is simply an encrypted user id + timestamp. 94.254.xxx.xxx - - [02/Jul/2011:22:25:46 ...
9
votes
1answer
587 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
7
votes
1answer
405 views

Should I ping to Google sitemap-index.xml or only the sitemap file that has been modified?

I have a sitemap-index.xml with 7 files. After updating one of the 7 with new content should I ping Google with the index file or with the updated file? In my index I also use lastmod so Google can ...
6
votes
3answers
901 views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
6
votes
1answer
93 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
6
votes
2answers
1k views

Google-bot sees “Sorry, we have no imagery here” on pages with Google Maps

I have a site with Google Maps on most of the pages. When inspecting content keywords in Google Webmaster tools, content keywords identified by Google-bot for the site include "imagery", "sorry" and ...
5
votes
5answers
967 views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
5
votes
3answers
480 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
5
votes
2answers
204 views

Robots.txt and pattern matching

Adding this to my robots.txt User-agent: * Disallow: /*action=*$ How does robots not recognizing wild cards handle this?
5
votes
5answers
573 views

Which tools can help limit maximum page view per IP to limit scrapers and bots?

I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
5
votes
1answer
119 views

How to interpret Google policy on crawling and indexing search results pages?

I just learned that we can be penalized for letting Google crawl and index our database search results. http://webmasters.stackexchange.com/a/55599/33777 Question: If they don't want to list Yellow ...
5
votes
1answer
2k views

link rel=“alternate”, multiple languages and canonical urls

Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain. On some ...
5
votes
1answer
594 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
5
votes
2answers
208 views

How frequently does Googlebot fetch sitemaps? Does it depend on PageRank?

How frequently does Google fetch sitemaps? I am now working with a high traffic website that normally has 30 new posts per minute. It currently it provides a sitemap which only includes the 100 ...
4
votes
2answers
146 views

When Googlebot sees a link, will it click it or navigate to it?

My site uses pushState and JSON data to display content. So, for example, this might appear on my page: <a href="/some/page">some page</a> The JavaScript then prevents the default ...
4
votes
1answer
246 views

Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?

I've seen code like for preventing Google from indexing part of a page: <p>This is a paragraph that will be indexed by Google.</p> <!--googleoff: index--> <p>This is a ...
4
votes
1answer
416 views

Does htpasswd authentication prevent Google from crawling my site?

I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing. I currently ...
4
votes
3answers
1k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
4
votes
2answers
65 views

Preventing Adsense Crawling Specific Regions of the Site

In my website, I wan't adsense crawler to not consider certain parts of the HTML. Is there any snippet I can use. The reason is I am using a plugin from an external provider and as soon as I installed ...
4
votes
2answers
851 views

Does duplicate content inside your own website matter for SEO? [duplicate]

Possible Duplicate: What is duplicate content and how can I avoid being penalized for it on my site? I'm not referring to the duplicate content where you copy content from another site and ...
4
votes
3answers
823 views

nofollow doesn't stop crawling?

In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" ...
4
votes
2answers
118 views

Google SEO - Migrating website from a sub-directory of another website to its own domain name

At the present moment I have a website hosted on example.com/myWebsite, where example.com hosts in its root directory a different website. I have the domain example.net, which redirects to ...
4
votes
1answer
221 views

Using rel=canonical and noindex in a 1-n partners environment

We sell complete sites including the domains to several partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original but the ...
3
votes
5answers
453 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
3
votes
1answer
36 views

Will Google follow link if anchor tag content is blocked?

This is a followup on a question I was asking here: Cropping images & SEO Basically, if I have something like this: <a href="<Full_Res_Image>"><img ...
3
votes
4answers
537 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
1answer
71 views

Relative URLs confuse Google crawler

I'm having some odd problems that are apparently being generated by Google crawler. Bing isn't these errors, so it's kind of confusing. The problem seems to be because of relative links. Google ...
3
votes
2answers
99 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
3
votes
1answer
156 views

Google is not indexing my site although it is crawling it

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message: URL and ...
3
votes
2answers
98 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
3
votes
1answer
243 views

Do Search Engine Spider's submit POST forms?

If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form? For example, if I were to have 4 ...
3
votes
2answers
102 views

Is offering Google Bot a lite version of site skin considered cloaking?

If you have multiple templates a user can choose from on a site (like a forum), and you offer Google Bot a "lite" skin, is this considered cloaking? Or is this considered okay? The actual content ...
3
votes
2answers
509 views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
3
votes
3answers
195 views

Why is the number of pages on my website indexed by Googlebot decreasing?

I am monitoring the number of pages indexed by Googlebot for one of my websites . I noticed that since last week, the number of pages has dropped by about 60 or so. Now to be fair, I have not added ...
3
votes
2answers
253 views

Foolproof way to ensure Google news pulls the correct image for its thumbnails?

Google news results have an acompanying thumbnail next to articles that show up in the results. If Google's crawler can't find a thumbnail to pull from our site, it uses its next best guess from ...
3
votes
1answer
55 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
3
votes
1answer
283 views

Can I use meta refresh instead of 301 redirect?

I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42. Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...
3
votes
1answer
184 views

Google 404 errors on webmaster page

I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
3
votes
1answer
834 views

IIS Dynamic IP Restrictions Module and Googlebot

I consider using IIS Dynamic IP Restrictions Module in order to make it hard for DoS (Denial of Service) attacks. But I am not sure how that would work out for search engine bots like Googlebot. Any ...
3
votes
5answers
376 views

SEO problems with multilanguage site [duplicate]

Possible Duplicate: How can I get search engines to crawl my site and see a localised view of my data? I have recently started a CMS project in PHP, the site will be multilingual. ...
3
votes
4answers
476 views

Personalized content (authenticated with a cookie) and impact on SEO

We have a big website with a lot of content, that is doing pretty well in terms of SEO. Google likes the site, visitors like it as well. Our problem is that there is a lot of content that would only ...
3
votes
1answer
48 views

Is there a way to stop a keyword from appearing in Google Webmaster Tools?

I added a web site a while ago to Google's index and it contains lots of links with the text "Permalink" that, well, link to permalinks of different content/pages. Now "Permalink" is the #1 keyword ...
3
votes
3answers
270 views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...
3
votes
1answer
149 views

Google crawlers screen size

Imagine a I have two types of images (A and AAA(heavier)) for the same website: depending on the size of the screen it will show one image or other. Now I'm wondering what images (A or AAA) loads ...
3
votes
1answer
446 views

Webmaster Tools show the “Total Indexed” number to be zero

I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
2
votes
4answers
548 views

Google bots are severely affecting site performance

I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on ...
2
votes
3answers
92 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
2
votes
1answer
123 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)