Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

2
votes
0answers
219 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
2
votes
1answer
734 views

Prevent Googlebot from following JavaScript redirect

I run a website called isitsnowingoutside.com as kind of an experimental project. As I am still learning- it is far from perfect. The basic premise is the site uses your location to determine if it is ...
2
votes
2answers
358 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
1answer
366 views

Google search : pubdate issue

At some point this year, Google Search started to show the publication date. When I updated my website, I noticed that the publication date was actually my birthday, simply because it's the first ...
2
votes
2answers
60 views

Google displays search results even after removing URLs in Search Console

I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" ...
2
votes
1answer
131 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
2answers
195 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
1
vote
5answers
3k views

How can I make Google crawl content behind an AJAX load more button

I have a feed that has 10 questions and answers loaded when you first open the page. And then I have a "Load More" button that gets you 10 new questions and answers everytime you click on it. How can ...
1
vote
3answers
227 views

How does Googlebot crawl a webpage? [closed]

Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
1
vote
4answers
2k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
2answers
82 views

Why can't Google Bot access my site?

I repeatedly get an warning/error message from Google as follows: Googlebot can't access your site I have checked, edited, removed, replaced the robots.txt file on the site all to no avail. ...
1
vote
2answers
218 views

When will my old page stop appearing on Google? [duplicate]

Possible Duplicate: Old pages still appearing in Google? I recently bought a new address for my Blogger blog, from yannbane.blogspot.com to www.yannbane.com. However, www.yannbane.com ...
1
vote
2answers
265 views

sitemap and crawler

I am still newbie in the entire world of SEO. Let's take a look at my example. I have a website very similar to blog style site. There few patterns for urls, all of the are very simple and allowed to ...
1
vote
3answers
253 views

Is it bad to hide content for SEO?

I'm designing a mobile CSS file for my website. It normally has a top navigation bar, like in here, Stack Exchange's sites. In my mobile version, I plan to hide the topbar, and put a button, which ...
1
vote
2answers
417 views

Should I expect Search Engines (particularly Google) to crawl epub and mobi files

Some of our content we are publishing as epub and mobi eBook format files. Will search engines crawl and index these in the same way as PDFs currently are. If the answer is yes, could you point me to ...
1
vote
2answers
4k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
1
vote
1answer
152 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
1
vote
1answer
336 views

hide some area (sidebar/footer) from googlebot. causing any SEO problem?

For example, I have a product page http://www.myshop.com/product/iphone.html in this page, normal users will see: content area (about the iPhone itself) sidebar links (links to other products: ...
1
vote
1answer
63 views

Why is this mobile/image Googlebot crawl rate so fast? (10-30 req per sec)

This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone ...
1
vote
2answers
48 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?
1
vote
2answers
60 views

Why can't I find a certain text on my site in Google?

I can't find certain text on my site in Google, even if I'm searching specifically on the page that this text in on: site:mysite.com/page.html hidden text I tried searching just for ...
1
vote
3answers
269 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
1
vote
3answers
912 views

Is this really GoogleBot, or is this a spammer?

I found this IP in my access log posting repeatedly thousands of times. Is this really GoogleBot, or is it a spammer? I hesitate to block it. 185.62.188.98 - - [13/May/2015:18:54:33 -0400] "POST ...
1
vote
2answers
716 views

Should I add the WWW version of my site to Google Webmaster Tools or Non-WWW or both?

I've found here similar topics but not exactly similar and without satisfying references - I'm going to launch a new domain, I understood it's better to have one pattern - whether www or non-www, so ...
1
vote
1answer
197 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
1
vote
1answer
174 views

Google attempting crawl of URLs with parameter randomly placed in URL

We have a weird SEO issue; In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors. There are a set of several hundred dynamic pages which we need to maintain (though not ...
1
vote
3answers
177 views

New or same URL on reactivation of user generated content?

We run a classified ad website. Our users can temporarily deactivate their classified ads. From a SEO point of view, when they reactivate it, should the ad use the same URL or use a new one? Each ...
1
vote
2answers
117 views

I don't think my site is being detected/crawled properly/having backlinks, and it's ruining SEO [closed]

This is really frustrating. I'm setting up a computer repairs business called Darley Tech. I have listed my site on many online directories to encourage backlinks. I have relevant and useful ...
1
vote
1answer
213 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
1
vote
2answers
111 views

Is using Progressive enhancement, overriding links with .js considered cloaking and can it be see by search engines?

I'm using .load to load content from a page /about.html, so that search engines can see what is going on. I've made the links that trigger the .load <a> tags to the page /about.html but for ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
1
vote
1answer
50 views

Google Master Tool: How allow external links in robots.txt

I have a problem with google rendering. Googlebot couldn't get all resources for this page. Because it's blocked by its own google robots.txt file. Here's an example of an external link. ...
1
vote
2answers
44 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
1answer
57 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
1
vote
1answer
57 views

Session ID Creating 404 Errors

Problem: Website contains URLs with parameters Session ID gets inserted into URLs when Googlebot crawls page with parameter (sorting) options Session ID is created and then expires, creating a 404 ...
1
vote
1answer
156 views

How to serve for International Googlebots if we redirect users to subdomains for language matching?

So, we have a site and a few sub-domains with content in different languages and we're using Hreflang and canonical implementations for SEO. We do detect the language that users use in their browsers ...
1
vote
2answers
47 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
1
vote
1answer
129 views

Fetch as Google Bot not working for `www.` version of site but is for `non-www.` version - Webmaster Tools

Ive recently acquired a new domain that i want to point at our existing domain, ive setup hosting for the new domain and then done a 301 redirect in the .htaccess file of the new domain to my existing ...
1
vote
1answer
1k views

Webmaster Tools “Fetch as Google” doesn't fetch my js/css/images on a CDN

I've discovered that when I do Google Webmaster Tools --> Fetch as Google on a page on my site (let's call it http://example.com), it shows the static assets (js, css, and images) as Temporarily ...
1
vote
1answer
28 views

Changing headers for small screens - SEO implications

I'm changing some of my pages to better display in different devices (tables, smart phones etc). I want to figure out a good way of handling headings. So far my options are hide headings for small ...
1
vote
2answers
86 views

Would it hurt my rankings to have the same URL display similar content depending on the visitor's country?

I manage a website that has different Web pages that use the same URL structure, and I don't use = & ? But depending on your location we will show you different links. Let say you are in the UK ...
1
vote
1answer
121 views

What URL with French words does Google like?

We are looking for an answer to "what to do with French URL's?" We are going to generate a lot of French URL's this include characters like: L'test tést thîs is a test So we have a few ...
1
vote
2answers
450 views

Crawler errors for unused URLs containing “mozekcdn-a.akamaihd.net”

In the AdSense dashboard, Google is reporting crawl errors for unexpected URLs. It shows that Google crawler was unable to access some of my 87 pages. When I checked those pages I found that I haven't ...
1
vote
2answers
85 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
1
vote
2answers
44 views

Can Googlebot read any hosting stats?

When Google Analytics isn't active for a website, can Googlebot (or any other Google entity) get any traffic stats about that website in any other way?
1
vote
1answer
57 views

Google not indexing translated content using url params

I have a website that uses a URL param (http://example.com?lang=es) to translate each page content. I know that this option is not recommended, but technically is the only one I can provide atm. I ...
1
vote
1answer
165 views

Google wont apply adsense policies for session based websites and Ajax loaded sites?

I have seen so many websites (Mostly User generated content - Getting contents from Third party API's) still displaying Adult like contents (with images) but still they are running well with Google ...
1
vote
1answer
43 views

How to avoid google from indexing like example.com/post/post/post/1-foobar.html?

I have a weird anomaly with one project. Where Google is indexing sites like: example.com/post/post/post/1-foobar.html While it should do it like: example.com/post/1-foobar.html I just ...
1
vote
1answer
357 views

Does a jQuery “rel=nofollow” even work?

It's a fairly common practice for webmasters to rel=nofollow external links, and I frequently see it done with jQuery. The script typically looks something like this (off the cuff here): ...
1
vote
2answers
94 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...