Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

2
votes
1answer
35 views

Using Fetch as Google on AngularJS

I am working on an eCommerce site and I have been asked to look at the SEO implications of our product listing page. I have done all the necessary steps to get the webmaster tools working fine on one ...
1
vote
0answers
34 views

Webmaster Tools “Fetch as Google” doesn't fetch my js/css/images on a CDN

I've discovered that when I do Google Webmaster Tools --> Fetch as Google on a page on my site (let's call it http://example.com), it shows the static assets (js, css, and images) as Temporarily ...
4
votes
2answers
463 views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
1
vote
2answers
23 views

Can Googlebot crawl and index data shown on a map (powered by Google)?

I have a web service that includes mainly data on map. Can Google fetch these data (asynchronously loaded )? The solution I'm opting right now is to set the data URLs in a regular sitemap, but there ...
1
vote
1answer
38 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
3
votes
3answers
245 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my ...
1
vote
1answer
20 views

Changing headers for small screens - SEO implications

I'm changing some of my pages to better display in different devices (tables, smart phones etc). I want to figure out a good way of handling headings. So far my options are hide headings for small ...
1
vote
0answers
36 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
7
votes
4answers
176 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
7
votes
2answers
269 views

Base64 encoded images and availability of their metadata for Googlebot

If I embed an image into a page as an img-src with base64 data URI, are image's metadata (EXIF, IPTC, XMP) still available for Google's imagebot?
1
vote
2answers
97 views

Conflicting test results when using The Mobile-Friendly Test and PageSpeed Insights

Our website uses an open source product to detect user agent and based on that, it servers mobile theme or desktop theme to users. I know the exact difference between mobile-friendly test and ...
0
votes
0answers
16 views

Google Crawler constantly reports ~2-3% errors

For the last year and a half, I have a constant 2-3% error rate while crawling my web site. Types of errors are "Server error: No response". I've switched ISP, but errors persist. Debugging of ...
0
votes
0answers
13 views

Question About Traffic From Google And GEO Location OF Website

I want to know lets say a person who lives in South Wimbledon, UK, runs a website and then outsource the website work (like writing articles and posting and other site management work etc..) to a ...
4
votes
1answer
82 views

How to hide a JavaScript popup for Googlebot?

I want to display a JavaScript popup when visitors access my website, however I believe Google will decrease my rankings when they detect a popup. So, is there any way to hide a JavaScript popup for ...
0
votes
0answers
13 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
1
vote
2answers
89 views

How to let Google's crawler know about new Atom/RSS updates?

What is the right way of letting Google's crawler know about new Atom/RSS updates? The documentation about sitemap is clear about how to ping the crawler, but when it comes to Atom/RSS, there is ...
2
votes
3answers
158 views

Backlinks: How does a company's name as the anchor text help with SEO?

Often I see footer links like: SEO by COMPANY NAME How does this help with SEO other than helping a company rank for "Company Name"? When nearby words like "SEO by" get picked up by a search engine ...
0
votes
1answer
60 views

Does Googlebot see dynamically loaded content? [closed]

Our modelling agency website uses the Photolux Wordpress theme. Our front page uses the same masonry template that the demo site: Demo. The only difference is that we've modified the masonry elements ...
2
votes
1answer
124 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
2
votes
2answers
52 views

Using table tag to structure a web page decrease referencing quality? [duplicate]

I am student in web programing. I use massively <table> ... </table> to structure my web page. My professor once told me it could decrease the referencing quality for my websites, so I ...
2
votes
1answer
596 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
3
votes
2answers
154 views

Google crawl rate is climbing but still no indexing?

This might belong on another stack site but idk. Soooo.... A year ago I started two sites that I didn't want indexed by Google or any other search engine since both sites were being developed based on ...
2
votes
1answer
453 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
2
votes
2answers
152 views

Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?

My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
1
vote
2answers
45 views

Would it hurt my rankings to have the same URL display similar content depending on the visitor's country?

I manage a website that has different Web pages that use the same URL structure, and I don't use = & ? But depending on your location we will show you different links. Let say you are in the UK ...
2
votes
1answer
47 views

Google isn't indexing AJAX URLs even after fetching them with _escaped_fragment

I'm still in the process of making my site crawlable through Google even though it's all AJAX content. I got it where if the URL contains escaped_fragment_ it retrieves the right information form the ...
0
votes
1answer
52 views

How to see my site's problems with Google?

I recently made a website, which aims for the search query "Boun Schedule Planner" It was like 5th or 6th for a few days. But now it's not even in the first 3 pages. There are no warnings in ...
1
vote
1answer
38 views

How to tell googlebot crawl more page from my site? [duplicate]

I have a small local news site, every one increase hundreds of articles. I have set a crontab, list new articles' url into a sitemap by hour. for example, I have 24 sitemaps which is generated by ...
1
vote
1answer
44 views

What URL with French words does Google like?

We are looking for an answer to "what to do with French URL's?" We are going to generate a lot of French URL's this include characters like: L'test tést thîs is a test So we have a few ...
2
votes
2answers
103 views

URLs for dialogs from Bootstrap modals popups are getting indexed in search engines, but without nav they aren't good landing pages

On my side, I use bootstrap modals (aka dialogs / overlays) with remote content to display the detail view of some things (for example for detail view of user reviews/comments). The code looks like ...
0
votes
0answers
7 views

How to influence the info column in Google's search result? [duplicate]

Google often shows an info column next to the search results, if it recognizes your search query to be directly related to a "thing" they know. Example: You search for "beatles" and get a box with ...
1
vote
3answers
59 views

Will providing an internal graph allow the Google bot to index my site faster, and update it more often?

A programmer, but a very inexperienced webmaster here. I am building a social site that gets content generated literally every day. On top, the content of my already indexed pages changes quite often ...
1
vote
0answers
62 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
10
votes
4answers
1k views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
1
vote
0answers
81 views

I still get “Googlebot can't access your site” messages after removing a site from Google Webmaster Tools

I had a client who moved their site a while back to another developer. When they moved it they took down the site and put up a coming soon page ( which is currently what's there). Googlebot warned me ...
1
vote
2answers
505 views

Google Webmaster Tools is reporting crawl errors for URLs with “joomla” on a WordPress site

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
2answers
422 views

Crawler errors for unused URLs containing “mozekcdn-a.akamaihd.net”

In the AdSense dashboard, Google is reporting crawl errors for unexpected URLs. It shows that Google crawler was unable to access some of my 87 pages. When I checked those pages I found that I haven't ...
1
vote
1answer
107 views

Google Adsbot suddenly crawling odd pages

I have an error checker on my password reset page which tells me everytime someone comes to reset their password without a valid token. I have noticed within the last couple of days that the ...
0
votes
1answer
30 views

How to tell how long google thinks it has been since your website was updated?

I never really thought 'freshness' was a problem for my site until I recently used the silktide 'nibbler' website grading report / scanner. It listed my 'freshness' as a big old 0 and said my site ...
1
vote
1answer
212 views

I get General HTTP error: 404 not found when submiting sitemap?

I just bought 2 domain from domaindiscount24.com. I get an error: "General HTTP error: 404 not found" when submitting sitemap. Then, I use "fetch as googlebot" and my website URL, it showed "not ...
1
vote
1answer
95 views

Google wont apply adsense policies for session based websites and Ajax loaded sites?

I have seen so many websites (Mostly User generated content - Getting contents from Third party API's) still displaying Adult like contents (with images) but still they are running well with Google ...
2
votes
2answers
96 views

What's the smallest page size for Googlebot to not go “Soft 404”?

Google appears to have two documents for Soft 404: https://support.google.com/webmasters/answer/181708?hl=en&ctx=tltp https://support.google.com/webmasters/answer/2409443?ctx=MCE&ctx=S4 ...
1
vote
2answers
70 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
2
votes
3answers
2k views

How much time does Googlebot take to crawl the page after submitting it to index

I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority. I've a page linking to newly added jobs ...
5
votes
3answers
612 views

Do Search Engine Spider's submit POST forms?

If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form? For example, if I were to have 4 ...
7
votes
1answer
181 views

Does Googlebot support SPDY?

When SPDY is available for a site, does Googlebot use it? If so, sites which are SPDY enabled would appear faster to Googlebot.
1
vote
2answers
160 views

Google crawl rate dropped after activating CloudFront

Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user ...
5
votes
1answer
70 views

Is it expected behaviour that Google also crawls #! urls?

I'm a developer for an AJAX webapp and, following Google's specification for crawlable webapps, we use #! to indicate that it's a AJAX application such that we can serve a static page to Google ...
2
votes
3answers
768 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...