Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
2answers
173 views

Google crawl rate dropped after activating CloudFront

Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user ...
1
vote
1answer
453 views

What does Google use 173.194.90.xxx IP's for? Why is it not a googlebot.com pointer?

I've been noticing this new Google range hitting our feeds over the last couple weeks. It's bordering on abusing our resources with multiple hits in short timeframe. The 173.194.90.xxx IP's fall under ...
1
vote
1answer
28 views

What would prompt multiple web crawlers to scan a site?

I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's mobile bots and others. Is this just bad luck or could something ...
1
vote
1answer
130 views

Can Google crawlers/googlebot read CSS content in before and after elements?

I have a div and I am placing some text in it via content in :before and :after. Can Google crawlers read this? Example: div:after { content: 'This is from CSS'; display:block; } Working ...
1
vote
1answer
37 views

Is it possible to include news into Google News if there is a non-news content under the same section?

We want to include our news into Google News, and now performing research how to properly do this. At the moment we have the same base URL for news and non-news articles (analysis, interviews, etc.), ...
1
vote
3answers
106 views

When Google gives up recrawling 301 that led to 404?

I've transferred a domain and made a mistake in the redirects (the URL structure is identical). Even though they went to the new domain, the error caused a 404 when crawled by Google bot. 10 days ...
1
vote
1answer
67 views

Does Googlebot share geo-location information to better crawl geo-aware websites?

I'm developing a website that allows users to share their location via JavaScript. Based on that location, either the content of the current page will change, or it will redirect a user to a page with ...
1
vote
1answer
39 views

SEO - All navigation links or just relevant navigation links

I'm trying to decide what the best approach is for rendering the static HTML for the websites navigation links. I have two possible options 1) render all links for every page, and hide non-relevant ...
1
vote
1answer
68 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
1answer
248 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
1
vote
2answers
110 views

Regarding Google Index -> Remove URLs

I have a question regarding Google index -> Remove URLs from Google Webmaster Tools. Basically I have replaced my site completely but upon searching Google, it returned some old pages that didn't ...
1
vote
1answer
20 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
1
vote
3answers
78 views

Create a way for Googlebot to find the deep pages in my site when users would find these pages via site search

I have a website like Foursquare, where users search for business in a specific location. The users can click on markers inside the map to view details about the companies/stores, and these details ...
1
vote
2answers
25 views

Can Googlebot crawl and index data shown on a map (powered by Google)?

I have a web service that includes mainly data on map. Can Google fetch these data (asynchronously loaded )? The solution I'm opting right now is to set the data URLs in a regular sitemap, but there ...
1
vote
1answer
275 views

Can Google bots interpret hyphen separated words as a single keyword?

I have a french website which frequently includes the word rendez-vous. But, Google Webmaster Tools content keywords tells me the #1 keyword is rendez which is not what I want. Even worse, the word ...
1
vote
1answer
43 views

Forum is removed but still being visited by Googlebot

I've removed Simple Machine Forum (SMF) from my blog and from Google's index . I've even removed the sub-directory /forum from my blog techwayz.com and blocked it through robots.txt file. ...
1
vote
1answer
566 views

Do web crawlers/spiders index azure web sites?

For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is ...
1
vote
1answer
472 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
1
vote
3answers
1k views

What are the problems of httponly and secure cookies?

We would like to set: all our cookies (session id, etc.) as "httponly" all security (session token, "remember me", etc.) cookies as "secure" Except compatiblity issues, what are the know problems ...
1
vote
1answer
171 views

Will Google crawl a news/RSS aggregator site?

I know how to code. I know how to set up a news/RSS aggregator site. But I don't know if Google will crawl it. Will Google crawl such a site? The site will have a listing of all the aggregated news ...
1
vote
1answer
96 views

Why old (301) links stay on Google when breaking site down to multiple domains

Some background: We did have single site and single domain (let's call it mainsite.com) with product information, however things have changed since and product database has grown fast. So we decided ...
1
vote
3answers
336 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
1
vote
1answer
404 views

How to make image only website friendly to googlebot? [duplicate]

so I making this portfolio site for some artist. She draw all the menus and texts manually(on paper), then scanned, some "post processing" in photoshop and that's how the whole layout was created. ...
1
vote
2answers
509 views

Google Webmaster Tools is reporting crawl errors for URLs with “joomla” on a WordPress site

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
1answer
106 views

Bots and scripts [duplicate]

Possible Duplicate: Will AJAX-fetched material be invisible to search engines? Can bots understand what's going on in javascript or vbscript? Most of the social networking modules and ad ...
1
vote
1answer
536 views

Googlebot visit but no cache update - why?

I have made a new plain vanilla HTML website. I have been making regular modifications to it on an almost daily basis. The site is hosted by hostmonster and as part of their service they offer ...
1
vote
0answers
46 views

Google Webmaster Tools can't access robots.txt and the home page is “temporarily unreachable”

My site www.chennaionnet.com suddenly disappeared from Google search and Webmaster Tools says it can't access robots.txt. Fetch as Google says "temporarily unreachable" for robots.txt and even for the ...
1
vote
2answers
61 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
1
vote
0answers
36 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
1
vote
0answers
67 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
1
vote
0answers
15 views

Sudden decrease in pages crawled per day (from 20k/day to 300/day) [duplicate]

I've seen large drops before (like the drop mentioned in this question), but I've never seen a drop like this before and am concerned that something is seriously wrong with our site's relationship ...
1
vote
1answer
151 views

Is creating Ajax Google Snapshot with less content than original page considered cloaking?

I've read a few posts about Google AJAX snapshot creation and I'm still asking myself a few questions: I have a full AJAX web site with really a lot of pages which differ from one to the other with a ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
1
vote
0answers
74 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: ...
1
vote
0answers
81 views

I still get “Googlebot can't access your site” messages after removing a site from Google Webmaster Tools

I had a client who moved their site a while back to another developer. When they moved it they took down the site and put up a coming soon page ( which is currently what's there). Googlebot warned me ...
1
vote
0answers
74 views

Why does Google keep hitting only one MP3 file?

I have a bunch of Mp3 files on my site, and all if them are in the sitemap to allow search engines that log mp3 files to know about them. For some reason, every day or two, Google keeps hitting the ...
1
vote
0answers
12 views

How do google bots work for sites that are database driven? [duplicate]

I didn't see this question already asked but I also don't know if this may be called something in particular so my apologies. While learning SQL and PHP I was curious to know how do bots search a ...
1
vote
0answers
34 views

What resolution does Google use while crawling a site with responsive/adaptive design? [duplicate]

I use conditional loading and media queries, JS, and CSS techniques for building responsive/adaptive designed websites. The HTML output for these sites depends on the number of horizontal and ...
1
vote
0answers
13 views

How do services that make subsites (ie. new subdomains) get those sites indexed by Google and other search enginex? [duplicate]

Services like tumblr, github pages, quora blogs, etc. make new subdomains when a user signs up or makes a site. So the subdomains are like: blogname.tumblr.com , pagename.github.io, ...
1
vote
0answers
72 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
1
vote
1answer
116 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
1
vote
0answers
58 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
2answers
371 views

How can I tell GoogleBot that a subdirectory is now a subdomain?

I had about a million pages of a catalog indexed under a subdirectory, and now that's moved to a subdomain. GoogleBot is crawling each one of them and getting a 301 redirect to the new location. Even ...
1
vote
0answers
184 views

Site returning 404 header to Google, not sure why

A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by Google at all (which is the main issue but I ...
1
vote
2answers
171 views

Confused by Fetch as Googlebot behaviour

I have a site that I 'm trying to index and I am a bit confused by what I should expect to see in the result of the Fetch as Googlebot tool. The site uses hashbangs, so according to Google it should ...
1
vote
2answers
194 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
0
votes
3answers
165 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
0
votes
1answer
195 views

Googlebot following invalid URLs. How do I troubleshoot the problem?

I'm getting errors generated by Googlebot while following invalid URLs. Checking with Google Webmaster tools confirmed that the problem was the bot: I've checked the source code and the HTML ...
0
votes
2answers
276 views

Should I add the WWW version of my site to Google Webmaster Tools or Non-WWW or both?

I've found here similar topics but not exactly similar and without satisfying references - I'm going to launch a new domain, I understood it's better to have one pattern - whether www or non-www, so ...
0
votes
2answers
60 views

Lynx shows nothing but “loading” when opening a site

The site I help to administer appears very poorly in Google search results. While trying to tackle the issue from several angles I've tried to open the site in Lynx browser (the intention was to see ...