Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

3
votes
1answer
57 views

Odd/faulty page links being served to Googlebot

Yahoo recently bought Tumblr, where I write fairly long form scientific pieces. I noticed recently that a new script had been added to the end of pages, and it seems to be interacting badly with the ...
3
votes
1answer
73 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
3
votes
3answers
505 views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...
3
votes
1answer
339 views

Can I use meta refresh instead of 301 redirect?

I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42. Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...
3
votes
1answer
188 views

Google 404 errors on webmaster page

I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
3
votes
5answers
431 views

SEO problems with multilanguage site [duplicate]

Possible Duplicate: How can I get search engines to crawl my site and see a localised view of my data? I have recently started a CMS project in PHP, the site will be multilingual. ...
3
votes
4answers
536 views

Personalized content (authenticated with a cookie) and impact on SEO

We have a big website with a lot of content, that is doing pretty well in terms of SEO. Google likes the site, visitors like it as well. Our problem is that there is a lot of content that would only ...
3
votes
1answer
54 views

Is there a way to stop a keyword from appearing in Google Webmaster Tools?

I added a web site a while ago to Google's index and it contains lots of links with the text "Permalink" that, well, link to permalinks of different content/pages. Now "Permalink" is the #1 keyword ...
3
votes
2answers
156 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
3
votes
1answer
251 views

Google crawlers screen size

Imagine a I have two types of images (A and AAA(heavier)) for the same website: depending on the size of the screen it will show one image or other. Now I'm wondering what images (A or AAA) loads ...
3
votes
2answers
96 views

Directing crawlers to content in language per language sub-domain

I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
3
votes
1answer
474 views

Webmaster Tools show the “Total Indexed” number to be zero

I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
2
votes
4answers
701 views

Google bots are severely affecting site performance

I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on ...
2
votes
3answers
174 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
2
votes
1answer
282 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
2
votes
2answers
80 views

Will multilingual content on same page works to avoid content duplication

We have a website, which is having old/new vehicle deals for purchase. We are getting these data from third party website (legally) and all the pages are in English language. So, Here our issue is ...
2
votes
4answers
1k views

Does updating site too frequently hurt SEO rankings?

I recently updated my site, like 3-4 times in 2 months. At first everything was good, (i.e., Google was updating my pages' SERP content in 2-3 days). But now I did kind of a major change, I switched ...
2
votes
3answers
184 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
2
votes
1answer
41 views

Does Googlebot determine the value of a link by its position on the page?

It's well-known that the SEO value of a website basically depends on how much it has been referenced, linked, and shared. From Googlebot's point of view, is a link at the top of a page more valuable ...
2
votes
2answers
106 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
2
votes
2answers
961 views

What does Google do with indexed pages returning 403?

I guess that Google removes already indexed pages that are now returning the HTTP error 403. Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
2
votes
1answer
275 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
2
votes
2answers
960 views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
2
votes
1answer
46 views

How to present videos for Google Bot

i have a website which streams Flash Videos. For Google i have a special page due to using AJAX for normal Users (see articles on "_escaped_fragment_" for this). What i want to know is how i should ...
2
votes
2answers
529 views

Google webmaster Verification failed

I have a site created by Ruby on Rails. I had successfully verified the site with Google Webmaster Tools some months ago. However, recently Webmaster Tools has been asking me to verify my site. I ...
2
votes
1answer
81 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
2
votes
2answers
756 views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
2
votes
2answers
174 views

Googlebot algorithm for crawling?

Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?
2
votes
2answers
193 views

Get google to index JavaScript pages as separate pages

I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. ...
2
votes
1answer
2k views

Google on capital letters in url

I created one of my websites with the following url structure: http://domain.com/Animal/Pig http://domain.com/Pigs If I change the url map to lower case letters instead, will google reindex those ...
2
votes
1answer
333 views

Making JSON page which renders on document ready indexable

I have a page which is pretty much 99% on document ready generated by various JS files. It then gets and renders tons of JSON (I use tempo JS for rendering). It all happens on document ready. So ...
2
votes
2answers
3k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
2
votes
3answers
91 views

suspicious crawler activity

I'm noticing that I get accesses 66.249.66.198 - - [01/Jul/2011:17:13:46 +0200] "GET /img/clip.incubus.torrent.phtml HTTP/1.1" 404 143 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
2
votes
2answers
137 views

Backlinks: How does a company's name as the anchor text help with SEO?

Often I see footer links like: SEO by COMPANY NAME How does this help with SEO other than helping a company rank for "Company Name"? When nearby words like "SEO by" get picked up by a search engine ...
2
votes
3answers
363 views

Google Webmaster Tools reporting crawl error 404 (Not found) for a page which has never existed

Google Webmaster Tools is reporting a crawl error "404 (Not Found)" for an .html page which has never existed on a web site I developed. The page being reported is "tyuiabmog.html", which doesn't ...
2
votes
2answers
289 views

Googlebot requesting invalid URL

I have a web app which emails me exceptions automatically. This morning there was an error relating to a url: /Catalog/LiveCatalog?id=ylwpfqzts id is invalid (should be a guid) and caused an error ...
2
votes
1answer
39 views

Why my webpages body content is not taken into account to display in google SERP?

It's confusing for me that I have content on my webpages, however, when I search with the site: operator it is not taken into account. There is something like "Thank you for contacting us...." ...
2
votes
1answer
96 views

SEO: If another website uses a iframe from our website, would the links in the iframe appear to be coming from our website or from theirs?

we are creating a "app" on our website that allows affiliates to simply use a generated iframe on their website, which would produce a list of all our products (we sell event vouchers). The links ...
2
votes
2answers
71 views

How to convince Google to list HTTPS version of website? [duplicate]

Having finally configured HTTPS for my company's website, I'm wondering now if there are any steps I can or should take to convince Google's search results to list the site as an HTTPS link rather ...
2
votes
1answer
49 views

Google not indexing my Mediawiki's jpg files description pages

My website es.wikineos.com has been running for a few years with the MediaWiki Content Management System like the one they use at Wikipedia. Google seems to be indexing my site correctly at a quick ...
2
votes
1answer
39 views

I have a DoFollow backlink but I can't go directly to my site without a confirmation to leave it

I have a DoFollow backlink to a forum and I checked it. My concern is before I can reach my site using that link, there is a message: You're leaving the site, are you sure you wanted to continue? ...
2
votes
1answer
6k views

Is “Identity not verified” a problem? If so, how to solve it?

When I select the context menu item "View Page Info" on my site (using Chrome), I see "Identity not verified" Should I be: (a) concerned (b) angry (c) nonchalant ? If one of the first two, what ...
2
votes
1answer
162 views

New URL after doing redirect using 301

If we do a redirect 301, one my expired job to another URL in my site, Google will record this redirection. But what if this original job URL appears again say after 1 year. Will Google remember this ...
2
votes
2answers
476 views

Is there any benefit to put rel='nofollow' on mailto links?

Since bots have a fixed budget to spend to crawl websites on a regular basis, does it makes sense to mark mailto links with rel='nofollow' to make sure them don't spend time on these (especially if ...
2
votes
1answer
536 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
2
votes
1answer
274 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
2
votes
2answers
639 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
2
votes
2answers
89 views

Forbid content in google results summary

I put a professional directory project online since few weeks, Google has started to index some pages but i just noticed a flaw in the design of the pages. Each page have an adress, some content and ...
2
votes
1answer
985 views

How to tell Google to stop crawling my website via IP?

After a database server migration, I've noticed that GoogleBot started getting errors when trying to crawl my site. The reason seems to be that it is hitting my site via IP address (and due to my ...