Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

2
votes
4answers
707 views

Does updating site too frequently hurt SEO rankings?

I recently updated my site, like 3-4 times in 2 months. At first everything was good, (i.e., Google was updating my pages' SERP content in 2-3 days). But now I did kind of a major change, I switched ...
2
votes
3answers
182 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
2
votes
2answers
711 views

What does Google do with indexed pages returning 403?

I guess that Google removes already indexed pages that are now returning the HTTP error 403. Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
2
votes
1answer
244 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
2
votes
2answers
758 views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
2
votes
1answer
46 views

How to present videos for Google Bot

i have a website which streams Flash Videos. For Google i have a special page due to using AJAX for normal Users (see articles on "_escaped_fragment_" for this). What i want to know is how i should ...
2
votes
2answers
463 views

Google webmaster Verification failed

I have a site created by Ruby on Rails. I had successfully verified the site with Google Webmaster Tools some months ago. However, recently Webmaster Tools has been asking me to verify my site. I ...
2
votes
2answers
202 views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
2
votes
2answers
116 views

Googlebot algorithm for crawling?

Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?
2
votes
2answers
179 views

Get google to index JavaScript pages as separate pages

I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. ...
2
votes
1answer
1k views

Google on capital letters in url

I created one of my websites with the following url structure: http://domain.com/Animal/Pig http://domain.com/Pigs If I change the url map to lower case letters instead, will google reindex those ...
2
votes
1answer
317 views

Making JSON page which renders on document ready indexable

I have a page which is pretty much 99% on document ready generated by various JS files. It then gets and renders tons of JSON (I use tempo JS for rendering). It all happens on document ready. So ...
2
votes
2answers
3k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
2
votes
3answers
87 views

suspicious crawler activity

I'm noticing that I get accesses 66.249.66.198 - - [01/Jul/2011:17:13:46 +0200] "GET /img/clip.incubus.torrent.phtml HTTP/1.1" 404 143 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
2
votes
2answers
132 views

Backlinks: How does a company's name as the anchor text help with SEO?

Often I see footer links like: SEO by COMPANY NAME How does this help with SEO other than helping a company rank for "Company Name"? When nearby words like "SEO by" get picked up by a search engine ...
2
votes
3answers
352 views

Google Webmaster Tools reporting crawl error 404 (Not found) for a page which has never existed

Google Webmaster Tools is reporting a crawl error "404 (Not Found)" for an .html page which has never existed on a web site I developed. The page being reported is "tyuiabmog.html", which doesn't ...
2
votes
2answers
257 views

Googlebot requesting invalid URL

I have a web app which emails me exceptions automatically. This morning there was an error relating to a url: /Catalog/LiveCatalog?id=ylwpfqzts id is invalid (should be a guid) and caused an error ...
2
votes
1answer
36 views

I have a DoFollow backlink but I can't go directly to my site without a confirmation to leave it

I have a DoFollow backlink to a forum and I checked it. My concern is before I can reach my site using that link, there is a message: You're leaving the site, are you sure you wanted to continue? ...
2
votes
1answer
2k views

Is “Identity not verified” a problem? If so, how to solve it?

When I select the context menu item "View Page Info" on my site (using Chrome), I see "Identity not verified" Should I be: (a) concerned (b) angry (c) nonchalant ? If one of the first two, what ...
2
votes
1answer
110 views

New URL after doing redirect using 301

If we do a redirect 301, one my expired job to another URL in my site, Google will record this redirection. But what if this original job URL appears again say after 1 year. Will Google remember this ...
2
votes
2answers
220 views

Is there any benefit to put rel='nofollow' on mailto links?

Since bots have a fixed budget to spend to crawl websites on a regular basis, does it makes sense to mark mailto links with rel='nofollow' to make sure them don't spend time on these (especially if ...
2
votes
1answer
295 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
2
votes
1answer
167 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
2
votes
2answers
521 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
2
votes
1answer
176 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
2
votes
2answers
88 views

Forbid content in google results summary

I put a professional directory project online since few weeks, Google has started to index some pages but i just noticed a flaw in the design of the pages. Each page have an adress, some content and ...
2
votes
1answer
827 views

How to tell Google to stop crawling my website via IP?

After a database server migration, I've noticed that GoogleBot started getting errors when trying to crawl my site. The reason seems to be that it is hitting my site via IP address (and due to my ...
2
votes
2answers
70 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
2
votes
2answers
47 views

Region specific language in Google Search results?

I have a dynamic English language site. If a user views the site in the US (with a US IP address) some of the site's language is altered so that it make more sense to a US audience. Anywhere else, ...
2
votes
1answer
64 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
2
votes
1answer
187 views

Googlebot received Access Denied when crawling a directory

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory ...
2
votes
1answer
187 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
2
votes
1answer
95 views

How to have Google crawl pages that are only up for a week

We have flash sale product pages that are only up for 7 days and then go down. What's the best way to have google crawl these pages properly? The URL for these pages go up and down every week and are ...
2
votes
2answers
419 views

How can I block visitors located in the U.S. while still allowing the Googlebot?

Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason we redirect all our U.S. traffic to a specialized page. While we do have these ...
2
votes
1answer
322 views

How do I block a user-agent from Apache

How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all bots from Apache on my debian server, that have the ...
2
votes
1answer
251 views

Will Google crawl session based website

I have a website, it is split into 3 categories but using PHP its an all-in one kind of style. When a user chooses a category on the home page a session is set, this is then used to set the style and ...
2
votes
1answer
89 views

Can bots crawl through “checkbox” form elements which act as a filter for products?

For example, my site has a list of products, with a filter box which allows you to narrow by price, features, etc. My question is, can google crawl this form, see the url results (which is a bunch of ...
2
votes
1answer
267 views

Removing Google Webmaster URL Parameters

Some time before I have created a post for removing full site from Google Index, after following many advices I have removed large number of pages from Google Index but now facing another problem as ...
2
votes
1answer
69 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
1answer
62 views

multilingual domains both crawled and come up to the wrong audience

I have a domain with multilingual support through subdomains. Each language version is identical besides the titles which are translated. e.g.: domain.com ja.domain.com es.domain.com I'm ...
2
votes
2answers
186 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
1answer
324 views

Googlebot can't access my site when crawling from rootdomain

I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. ...
2
votes
1answer
369 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
103 views

Crawl Errors (404) Showing Up in WMT

I see from WMT that my site has 4 (404) crawl errors, each Linked from 2 separate pages. As this is a small directory listings site it would be difficult to delete the missing URLs from the db each ...
2
votes
1answer
105 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
2
votes
1answer
213 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
2
votes
2answers
83 views

Directing crawlers to content in language per language sub-domain

I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
2
votes
2answers
246 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
2
votes
1answer
277 views

Google search : pubdate issue

At some point this year, Google Search started to show the publication date. When I updated my website, I noticed that the publication date was actually my birthday, simply because it's the first ...