Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

3
votes
1answer
87 views

Why is the date displayed in the Google search result is 3 days older than the actual publication date?

Yesterday I published an article on my website, if I go on Google it says that it has been published 3 days ago. Why? I've WordPress, on a Centos 6,5 VM. System time is wrong of 20 min of my server, ...
1
vote
1answer
44 views

When will google report correct results with its fetch tool?

I visited Google Webmaster Tools today after resubmitting sitemaps for my website, and while google is happy to index more HTML pages of my site, it however has not indexed a single image yet. So I ...
4
votes
0answers
552 views

How Googlebot discovers new website? [duplicate]

I have purchased a domain and uploaded with HTML Page and When I search in Google it shows my website. Now my question is How Google will discover my new website? I have not used ...
0
votes
0answers
10 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
0
votes
1answer
33 views

Why does Google Cache break my iframes?

In HTML5, iframe comes as a replacement for frameset. However, when I load my iframe-based page in SeaMonkey through Google Cache, the page appears empty, other than the standard Google Cache header ...
2
votes
2answers
34 views

Does Google index images declared using 'srcset'?

As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
3
votes
1answer
50 views

Replace all images when search engine crawler visits site

I have written code that will detect when a user is a search engine and i dont include css files when this happens to save bandwidth. I was also thinking i could replace ALL images on the page with a ...
0
votes
1answer
28 views

Changing pagination, and having more sections per page

I'm upgrading an old website, from having 25 results per page (40 pages), to having 50-100 per page (5-10 pages) This will change the pagination, and maybe affect search engines in a bad way.. So I'm ...
1
vote
3answers
302 views

Is this really GoogleBot, or is this a spammer?

I found this IP in my access log posting repeatedly thousands of times. Is this really GoogleBot, or is it a spammer? I hesitate to block it. 185.62.188.98 - - [13/May/2015:18:54:33 -0400] "POST ...
0
votes
0answers
23 views

How post or page of facebook, twitter, forum or classified sites (HIGH PR) show instantally in google or other search engines

My question is about SEO. How post or page of facebook, twitter, forum or classified sites(HIGH PR) show instantally in google or other search engines. As i know google take few days for crawl. Then ...
0
votes
0answers
57 views

Googlebot follows rel=“nofollow” links

I would like to hide all pages of our website from being read or indexed by Googlebot, except several thousand of pages which were selected for indexing. The site was launched in April 2015. The ...
0
votes
0answers
45 views

google displays wrong language on multi language website because it takes cookies into consideration

I have a multiple language website that uses both subdomains and cookies to display the right language. Here is the logic on the website: de subdomain will display the German version fr subdomain ...
11
votes
4answers
2k views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
0
votes
0answers
12 views

Remove all pages from site [duplicate]

I have a staging server that accidentally got indexed by google. I want to remove ALL pages from google for that site: i.e. stagingserver.mycompany.com. I added a robots.txt to the root of the site ...
1
vote
1answer
71 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
3
votes
1answer
93 views

Does every page HAVE to be mobile friendly even if its almost unimportant?

I received the following regarding a URL of a page recommending a feature that only applies to desktop users: Faulty Redirect URLs redirect to an irrelevant landing page instead of the ...
0
votes
2answers
44 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
0
votes
2answers
45 views

Soft 404 for smartphones

We have redirected all non-existing pages to home page. It was fine for some days. Few days back i saw increase in the number of soft 404 errors for desktop. And the errors were the page urls that ...
1
vote
1answer
41 views

Fetch as Google Bot not working for `www.` version of site but is for `non-www.` version - Webmaster Tools

Ive recently acquired a new domain that i want to point at our existing domain, ive setup hosting for the new domain and then done a 301 redirect in the .htaccess file of the new domain to my existing ...
0
votes
1answer
13 views

Testing server indexed in Google [duplicate]

Assume our website domain to be www.example.com. And we have created a separate testing environment within our main domain: Say : www.project.example.com. So we have two versions within our main ...
1
vote
1answer
29 views

How does an adsense bot determine the “Above the fold” space on mobile and desktop sites?

I ran an experiment last night by replacing a small ad for a mobile site thats located near the top of the page with a larger box ad, but I moved that ad towards the bottom of the page. Ten hours ...
5
votes
3answers
124 views

For webpages using jQuery, will the Googlebot index them on “page load” or after the “page ready” state?

Will the Googlebot index a webpage on "page load" or after the "page ready" state. The reason I'm asking is because I'm trying to change my h1 and h2 tags using jQuery on page Load.
4
votes
3answers
182 views

How do I prevent Google from indexing text that is generated by JavaScript?

Google is executing the JavaScript on my password creator website and indexing the random passwords that get generated by the JavaScript. The random text that is generated could look spammy to ...
1
vote
0answers
9 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
4
votes
1answer
64 views

Does Google now factor the prominence of how a link is rendered into their PageRank algorithm?

Google's John Mueller now says that Googlebot renders the pages it crawls to determine what content is on them. They do this because so many sites are now heavily JavaScript centric and this allows ...
5
votes
3answers
3k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
1
vote
2answers
41 views

Duplicate Sitemap URLs

I'm in the process of replacing my old sitemaps with a new form which should better represent our site. However, as bots can take days or even weeks to scan all of our urls after submitted I do not ...
6
votes
3answers
735 views

Do Search Engine Spider's submit POST forms?

If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form? For example, if I were to have 4 ...
3
votes
3answers
78 views

Facets: Googlebot encountered extremely large numbers of links on your site

We are encountering the following issue: Googlebot encountered extremely large numbers of links on your site. The site is an ecommerce site and has around 12M pages indexed in Google. From the ...
6
votes
2answers
1k views

Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?

I've seen code like for preventing Google from indexing part of a page: <p>This is a paragraph that will be indexed by Google.</p> <!--googleoff: index--> <p>This is a ...
3
votes
1answer
158 views

Webmaster Tools shows URL errors linked from http://com.example.www/:http

I have a bog standard WordPress 4.2.1 site. I recently added it to Webmaster Tools and it is showing me URL errors (404s) that are linked from a weird URL: http://com.example.www/:http (I have ...
1
vote
0answers
126 views

Google Webmaster Tools can't access robots.txt and the home page is “temporarily unreachable”

My site http://goo.gl/xH2Ffy suddenly disappeared from Google search and Webmaster Tools says it can't access robots.txt. Fetch as Google says "temporarily unreachable" for robots.txt and even for the ...
2
votes
1answer
41 views

rel=“prev” and rel=“next” inserting using javascript

does adding the rel="next" tag after the page is loaded with a small javascript still useful for SEO and google? i have a problem in my code, basically the total number of pages in the catalog is ...
0
votes
1answer
61 views

Am I doing .htaccess right to block google, bing and alexa bots?

I do not need bots at all. I bought my traffic and I do not want my sites to show up on search engine or anything. So I added this: <IfModule mod_setenvif.c> # remove most popular bot ...
4
votes
4answers
833 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
0answers
9 views

Block search engine from indexing Currency Words as content keywords [duplicate]

I have one website with currency converter on it and Google WMT is considering the currency words as Content keywords. Is there anyway to block search engine from indexing this Currency Words like ...
1
vote
2answers
104 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
1
vote
1answer
359 views

Webmaster Tools “Fetch as Google” doesn't fetch my js/css/images on a CDN

I've discovered that when I do Google Webmaster Tools --> Fetch as Google on a page on my site (let's call it http://example.com), it shows the static assets (js, css, and images) as Temporarily ...
2
votes
1answer
219 views

Using Fetch as Google on AngularJS

I am working on an eCommerce site and I have been asked to look at the SEO implications of our product listing page. I have done all the necessary steps to get the webmaster tools working fine on one ...
5
votes
2answers
1k views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
1
vote
2answers
54 views

Can Googlebot crawl and index data shown on a map (powered by Google)?

I have a web service that includes mainly data on map. Can Google fetch these data (asynchronously loaded )? The solution I'm opting right now is to set the data URLs in a regular sitemap, but there ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
3
votes
3answers
374 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my ...
1
vote
1answer
23 views

Changing headers for small screens - SEO implications

I'm changing some of my pages to better display in different devices (tables, smart phones etc). I want to figure out a good way of handling headings. So far my options are hide headings for small ...
1
vote
0answers
47 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
8
votes
4answers
635 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
7
votes
2answers
394 views

Base64 encoded images and availability of their metadata for Googlebot

If I embed an image into a page as an img-src with base64 data URI, are image's metadata (EXIF, IPTC, XMP) still available for Google's imagebot?
1
vote
2answers
246 views

Conflicting test results when using The Mobile-Friendly Test and PageSpeed Insights

Our website uses an open source product to detect user agent and based on that, it servers mobile theme or desktop theme to users. I know the exact difference between mobile-friendly test and ...
0
votes
0answers
21 views

Google Crawler constantly reports ~2-3% errors

For the last year and a half, I have a constant 2-3% error rate while crawling my web site. Types of errors are "Server error: No response". I've switched ISP, but errors persist. Debugging of ...
0
votes
0answers
20 views

Question About Traffic From Google And GEO Location OF Website

I want to know lets say a person who lives in South Wimbledon, UK, runs a website and then outsource the website work (like writing articles and posting and other site management work etc..) to a ...