Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

11
votes
5answers
4k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
11
votes
2answers
815 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
9
votes
2answers
320 views

How is Googlebot finding URLs that are only visible to authenticated users?

Here is one of my customers, performing some action after having logged in to his account. The unique token is simply an encrypted user id + timestamp. 94.254.xxx.xxx - - [02/Jul/2011:22:25:46 ...
9
votes
1answer
602 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
8
votes
3answers
1k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
8
votes
2answers
71 views

Informing Googlebot for deprecated pages

I publish timetables in my website. For example last year I published Number 2 bus Summer 2013 timetable. I has pretty good ranking on Google SERPs for number 2 bus timetable But this year I added a ...
7
votes
1answer
421 views

Should I ping to Google sitemap-index.xml or only the sitemap file that has been modified?

I have a sitemap-index.xml with 7 files. After updating one of the 7 with new content should I ping Google with the index file or with the updated file? In my index I also use lastmod so Google can ...
7
votes
2answers
2k views

Google-bot sees “Sorry, we have no imagery here” on pages with Google Maps

I have a site with Google Maps on most of the pages. When inspecting content keywords in Google Webmaster tools, content keywords identified by Google-bot for the site include "imagery", "sorry" and ...
6
votes
4answers
162 views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
6
votes
1answer
192 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
6
votes
1answer
2k views

link rel=“alternate”, multiple languages and canonical urls

Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain. On some ...
6
votes
2answers
83 views

Does Googlebot support TLS v1.2? If not, what will my TLS v1.2-only site look like to it?

I currently redirect all non-HTTPS connections to HTTPS, and support TLS v1, TLS v1.1, and TLS v1.2. I reconfigured my testing server to accept only a very limited and secure cipher suite and TLSv1.2 ...
6
votes
2answers
219 views

How frequently does Googlebot fetch sitemaps? Does it depend on PageRank?

How frequently does Google fetch sitemaps? I am now working with a high traffic website that normally has 30 new posts per minute. It currently it provides a sitemap which only includes the 100 ...
5
votes
5answers
1k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
5
votes
3answers
509 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
5
votes
2answers
224 views

Robots.txt and pattern matching

Adding this to my robots.txt User-agent: * Disallow: /*action=*$ How does robots not recognizing wild cards handle this?
5
votes
3answers
1k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
5
votes
5answers
779 views

Which tools can help limit maximum page view per IP to limit scrapers and bots?

I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
5
votes
1answer
164 views

How to interpret Google policy on crawling and indexing search results pages?

I just learned that we can be penalized for letting Google crawl and index our database search results. http://webmasters.stackexchange.com/a/55599/33777 Question: If they don't want to list Yellow ...
5
votes
1answer
618 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
5
votes
1answer
91 views

How to deal with non-relevant keywords in Google Webmaster Tools

Google webmaster tools is showing keywords such as "cookies" in my keyword list. This is probably because we have links to our long legal disclaimer on cookies. We're a B2B service, so clearly I ...
4
votes
2answers
170 views

When Googlebot sees a link, will it click it or navigate to it?

My site uses pushState and JSON data to display content. So, for example, this might appear on my page: <a href="/some/page">some page</a> The JavaScript then prevents the default ...
4
votes
2answers
581 views

Hijacked meta description

On my site: faithhopeandfiction someone hacked into my website and now it's displaying wrong Meta description on Google search results. I've verified this using "view source" that meta description and ...
4
votes
1answer
381 views

Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?

I've seen code like for preventing Google from indexing part of a page: <p>This is a paragraph that will be indexed by Google.</p> <!--googleoff: index--> <p>This is a ...
4
votes
1answer
477 views

Does htpasswd authentication prevent Google from crawling my site?

I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing. I currently ...
4
votes
3answers
139 views

Bingbot requests from Google IP address

We have some suspicious requests to our server, 74.125.186.46 - - [24/Aug/2014:23:24:11 -0500] "GET <url> HTTP/1.1" 200 16912 "-" "Mozilla/5.0 (compatible; bingbot/2.0; ...
4
votes
5answers
141 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
4
votes
2answers
67 views

Preventing Adsense Crawling Specific Regions of the Site

In my website, I wan't adsense crawler to not consider certain parts of the HTML. Is there any snippet I can use. The reason is I am using a plugin from an external provider and as soon as I installed ...
4
votes
2answers
900 views

Does duplicate content inside your own website matter for SEO? [duplicate]

Possible Duplicate: What is duplicate content and how can I avoid being penalized for it on my site? I'm not referring to the duplicate content where you copy content from another site and ...
4
votes
1answer
1k views

IIS Dynamic IP Restrictions Module and Googlebot

I consider using IIS Dynamic IP Restrictions Module in order to make it hard for DoS (Denial of Service) attacks. But I am not sure how that would work out for search engine bots like Googlebot. Any ...
4
votes
3answers
877 views

nofollow doesn't stop crawling?

In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" ...
4
votes
2answers
123 views

Google SEO - Migrating website from a sub-directory of another website to its own domain name

At the present moment I have a website hosted on example.com/myWebsite, where example.com hosts in its root directory a different website. I have the domain example.net, which redirects to ...
4
votes
1answer
270 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
4
votes
1answer
227 views

Using rel=canonical and noindex in a 1-n partners environment

We sell complete sites including the domains to several partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original but the ...
3
votes
5answers
461 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
3
votes
1answer
46 views

Will Google follow link if anchor tag content is blocked?

This is a followup on a question I was asking here: Cropping images & SEO Basically, if I have something like this: <a href="<Full_Res_Image>"><img ...
3
votes
2answers
128 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
3
votes
4answers
607 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
1answer
116 views

Relative URLs confuse Google crawler

I'm having some odd problems that are apparently being generated by Google crawler. Bing isn't these errors, so it's kind of confusing. The problem seems to be because of relative links. Google ...
3
votes
2answers
145 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
3
votes
1answer
217 views

Google is not indexing my site although it is crawling it

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message: URL and ...
3
votes
1answer
359 views

Do Search Engine Spider's submit POST forms?

If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form? For example, if I were to have 4 ...
3
votes
2answers
110 views

Is offering Google Bot a lite version of site skin considered cloaking?

If you have multiple templates a user can choose from on a site (like a forum), and you offer Google Bot a "lite" skin, is this considered cloaking? Or is this considered okay? The actual content ...
3
votes
2answers
616 views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
3
votes
3answers
216 views

Why is the number of pages on my website indexed by Googlebot decreasing?

I am monitoring the number of pages indexed by Googlebot for one of my websites . I noticed that since last week, the number of pages has dropped by about 60 or so. Now to be fair, I have not added ...
3
votes
3answers
45 views

Indexed webpages stopped rising, while Submited keep rising normally

Last months I have noticed a strange behaviour in Google's Webmaster Tools. Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while ...
3
votes
2answers
417 views

Foolproof way to ensure Google news pulls the correct image for its thumbnails?

Google news results have an acompanying thumbnail next to articles that show up in the results. If Google's crawler can't find a thumbnail to pull from our site, it uses its next best guess from ...
3
votes
1answer
45 views

Odd/faulty page links being served to Googlebot

Yahoo recently bought Tumblr, where I write fairly long form scientific pieces. I noticed recently that a new script had been added to the end of pages, and it seems to be interacting badly with the ...
3
votes
1answer
68 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
3
votes
3answers
403 views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...