Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
2answers
128 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
1
vote
3answers
379 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
5answers
1k views

How can I make Google crawl content behind an AJAX load more button

I have a feed that has 10 questions and answers loaded when you first open the page. And then I have a "Load More" button that gets you 10 new questions and answers everytime you click on it. How can ...
1
vote
3answers
211 views

How does Googlebot crawl a webpage? [closed]

Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
1
vote
4answers
1k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
2answers
206 views

When will my old page stop appearing on Google? [duplicate]

Possible Duplicate: Old pages still appearing in Google? I recently bought a new address for my Blogger blog, from yannbane.blogspot.com to www.yannbane.com. However, www.yannbane.com ...
1
vote
2answers
248 views

sitemap and crawler

I am still newbie in the entire world of SEO. Let's take a look at my example. I have a website very similar to blog style site. There few patterns for urls, all of the are very simple and allowed to ...
1
vote
3answers
183 views

Is it bad to hide content for SEO?

I'm designing a mobile CSS file for my website. It normally has a top navigation bar, like in here, Stack Exchange's sites. In my mobile version, I plan to hide the topbar, and put a button, which ...
1
vote
2answers
689 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
2answers
295 views

Should I expect Search Engines (particularly Google) to crawl epub and mobi files

Some of our content we are publishing as epub and mobi eBook format files. Will search engines crawl and index these in the same way as PDFs currently are. If the answer is yes, could you point me to ...
1
vote
1answer
149 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
1
vote
1answer
296 views

hide some area (sidebar/footer) from googlebot. causing any SEO problem?

For example, I have a product page http://www.myshop.com/product/iphone.html in this page, normal users will see: content area (about the iPhone itself) sidebar links (links to other products: ...
1
vote
2answers
69 views

Is this really GoogleBot, or is this a spammer?

I found this IP in my access log posting repeatedly thousands of times. Is this really GoogleBot, or is it a spammer? I hesitate to block it. 185.62.188.98 - - [13/May/2015:18:54:33 -0400] "POST ...
1
vote
1answer
127 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
1
vote
1answer
133 views

Google attempting crawl of URLs with parameter randomly placed in URL

We have a weird SEO issue; In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors. There are a set of several hundred dynamic pages which we need to maintain (though not ...
1
vote
3answers
165 views

New or same URL on reactivation of user generated content?

We run a classified ad website. Our users can temporarily deactivate their classified ads. From a SEO point of view, when they reactivate it, should the ad use the same URL or use a new one? Each ...
1
vote
1answer
425 views

Stop Google crawling links completely

I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told ...
1
vote
2answers
113 views

I don't think my site is being detected/crawled properly/having backlinks, and it's ruining SEO [closed]

This is really frustrating. I'm setting up a computer repairs business called Darley Tech. I have listed my site on many online directories to encourage backlinks. I have relevant and useful ...
1
vote
1answer
180 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
1
vote
2answers
102 views

Is using Progressive enhancement, overriding links with .js considered cloaking and can it be see by search engines?

I'm using .load to load content from a page /about.html, so that search engines can see what is going on. I've made the links that trigger the .load <a> tags to the page /about.html but for ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
1
vote
1answer
133 views

Webmaster Tools “Fetch as Google” doesn't fetch my js/css/images on a CDN

I've discovered that when I do Google Webmaster Tools --> Fetch as Google on a page on my site (let's call it http://example.com), it shows the static assets (js, css, and images) as Temporarily ...
1
vote
1answer
21 views

Changing headers for small screens - SEO implications

I'm changing some of my pages to better display in different devices (tables, smart phones etc). I want to figure out a good way of handling headings. So far my options are hide headings for small ...
1
vote
2answers
48 views

Would it hurt my rankings to have the same URL display similar content depending on the visitor's country?

I manage a website that has different Web pages that use the same URL structure, and I don't use = & ? But depending on your location we will show you different links. Let say you are in the UK ...
1
vote
1answer
52 views

What URL with French words does Google like?

We are looking for an answer to "what to do with French URL's?" We are going to generate a lot of French URL's this include characters like: L'test tést thîs is a test So we have a few ...
1
vote
2answers
429 views

Crawler errors for unused URLs containing “mozekcdn-a.akamaihd.net”

In the AdSense dashboard, Google is reporting crawl errors for unexpected URLs. It shows that Google crawler was unable to access some of my 87 pages. When I checked those pages I found that I haven't ...
1
vote
2answers
71 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
1
vote
2answers
42 views

Can Googlebot read any hosting stats?

When Google Analytics isn't active for a website, can Googlebot (or any other Google entity) get any traffic stats about that website in any other way?
1
vote
1answer
38 views

Google not indexing translated content using url params

I have a website that uses a URL param (http://example.com?lang=es) to translate each page content. I know that this option is not recommended, but technically is the only one I can provide atm. I ...
1
vote
1answer
101 views

Google wont apply adsense policies for session based websites and Ajax loaded sites?

I have seen so many websites (Mostly User generated content - Getting contents from Third party API's) still displaying Adult like contents (with images) but still they are running well with Google ...
1
vote
1answer
35 views

How to avoid google from indexing like example.com/post/post/post/1-foobar.html?

I have a weird anomaly with one project. Where Google is indexing sites like: example.com/post/post/post/1-foobar.html While it should do it like: example.com/post/1-foobar.html I just ...
1
vote
1answer
135 views

Does a jQuery “rel=nofollow” even work?

It's a fairly common practice for webmasters to rel=nofollow external links, and I frequently see it done with jQuery. The script typically looks something like this (off the cuff here): ...
1
vote
2answers
66 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...
1
vote
1answer
232 views

Does Fetch as Googlebot still support their ajax-crawling proposal?

I spent half a day implementing the server side html generation for modal pages based on their proposal (link), but it seems like the Fetch as Googlebot functionality in Webmaster tools completely ...
1
vote
2answers
293 views

SEO penalty for 301 redirecting googlebot only?

I have a website that is splitting into two. For the sake of easily describing the what and why, let's say I run a classifieds site and I want to move all the adult content to its own domain. (very ...
1
vote
3answers
37 views

Older post not indexed, new post indexed right away?

I have a post I created yesterday that was indexed rather quickly by Google. However, I have two other post that I created within the past 4 days that are not indexed and I cannot find a trace of ...
1
vote
1answer
205 views

Subdomain and Googlebot can't access your site

I have a subdomain and Google Webmaster Tools, told me this: Googlebot can't access your site This is not the first time that I receive this message from Google's Webmaster Tools. I need to ...
1
vote
1answer
249 views

Creating sitemap for Googlebot - how to mark dynamic content / dynamic subpages?

I have a website that is an Internet forum. This forum has many categories, and a single category page that contains a lot of subpages with listed threads. This Internet forum is brand new, and about ...
1
vote
2answers
104 views

Somehow Google considers a properly 301'd URL as 200 and is still indexing the new content in old page?

We redirected all the old URL's to new ones properly using .htaccess. The problem is Google, somehow is still finding content in the old page (which it shouldn't) and stores it in the cache rather ...
1
vote
1answer
198 views

Will Google index a PHP header redirect?

I have a new domain which has an index.php file. Inside this file, I get user's IP, save it and redirect to another page on another domain. So there is no output, only header info. Will Google index ...
1
vote
1answer
194 views

Google visits on Statcounter. How?

In my website’s Statcounter report, I found page views from Google. I thought it may be a Google bot’s visit but I found that particular visit comes through a search query (example.com+keyword) and it ...
1
vote
2answers
114 views

Does specifying full webpage path in robots.txt would affect my website?

It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some label path of my website using robots.txt file. User-agent: * Disallow: ...
1
vote
1answer
564 views

Googlebot getting “301 Moved Permanently” according to “Fetch as Googlebot” in Webmaster Tools

I launched a website four days ago, and now when I try to crawl the website like Google bot in "Google webmaster tools" I end up with a response like this: HTTP/1.1 301 Moved Permanently Date: Thu, ...
1
vote
2answers
438 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
1
vote
3answers
3k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
1
vote
1answer
137 views

Could this be considered duplicate content? [duplicate]

Possible Duplicate: What is duplicate content and how can I avoid being penalized for it on my site? I'm building a dating site at the moment. On the homepage I have a paginated list of all ...
1
vote
1answer
325 views

Is Googlebot case sensitive? Should I use a 301 redirect to choose a preferred URL case?

The Googlebot reports the following: Pages with duplicate meta descriptions /Default.aspx?p=2 /default.aspx?p=2 Should I use a 301 redirect from Default to default or vice versa? Is Googlebot ...
1
vote
1answer
3k views

How to fix Google 404 not found Crawl Errors?

I was checking on Google webmater tool for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
1
vote
1answer
817 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
1
vote
1answer
105 views

Googlebot fetching lots of URLs with a “workaround” parameter

A certain page on our website keeps generating entries like the following on our IIS6 logs: 2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707368055555 80 - 66.249.85.65 - ...