Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

4
votes
2answers
472 views

Is it bad for SEO to submit a partially finished site to Google?

Hypothetical Situation 1: I have a plan for an amazing large-scale website with lot's of really cool features, animations and interactivity. This will take a several months to completely develop. ...
4
votes
2answers
587 views

Hide joyride (intro javascript) for Googlebot

I am using a joyride script for new visitors. Example here: http://foundation.zurb.com/sites/docs/v/5.5.3/components/joyride.html After the vistior is done with the intro, a cookie will be created ...
4
votes
2answers
37 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
1
vote
2answers
58 views

Why can't I find a certain text on my site in Google?

I can't find certain text on my site in Google, even if I'm searching specifically on the page that this text in on: site:mysite.com/page.html hidden text I tried searching just for ...
1
vote
2answers
59 views

How Google crawler behaves when the HTML page is 1.6MB? [duplicate]

Is it possible that Google crawler doesn't fetch all the content on the page when it's size is 1.6 MB? How does Google treat big documents?
2
votes
2answers
42 views

Will googlebot misinterpret my site if I had no spacing between anchor tags?

Some sections of my website involve displaying a list of photo thumbnails. The source code to that is similar to the following when condensed: <div> <a href="1">1</a> <a ...
1
vote
1answer
52 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
1
vote
1answer
98 views

How to update the date on Google search results?

For a blog entry, I have this meta tags: <meta property="article:published_time" content="2015-09-07T21:54:43+00:00" /> <meta property="article:modified_time" ...
2
votes
2answers
129 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
1
vote
2answers
82 views

Why can't Google Bot access my site?

I repeatedly get an warning/error message from Google as follows: Googlebot can't access your site I have checked, edited, removed, replaced the robots.txt file on the site all to no avail. ...
0
votes
0answers
202 views

SEO on ui-router states of AngularJS SPA

I have a simple SPA created with AngularJS, you can see it running here: www.consami.com.br. I was using Prerender.io to load a pure HTML version for crawlers, however some pages were not indexed by ...
0
votes
0answers
38 views

Crawl Budget economy: what to do with URLs with parameters?

So, a site has inbound links to a lot of important pages and these links have URL parameters. These URL parameters are basically used just for tracking purposes, similar to the way Amazon links to its ...
0
votes
0answers
32 views

access.log requests from UA “Google favicon”…? What do these mean?

I have a bunch of these in my nginx log file, on a range of URLs: on 66.102.9.23 - - [12/Nov/2015:10:40:25 +0000] "GET /myadminfolder/page.html HTTP/1.1" 200 1247 1800 "-" "Google favicon" ...
0
votes
0answers
24 views

How should I render a widget with many links inside a blog so google don't penalize the blog?

We run a service (will not mention the URL, as this is not a pitch) and need advice on Google-viewing-links matters. The service is for bloggers and allows them to create a "linkup" inside their blog ...
0
votes
0answers
13 views

Google not crawling all pages [duplicate]

I have over 200,000 pages but only about 23,000 pages are crawled by Google as is evident when i search for site:mywebsite.com and the result comes as About 22,900 results (0.18 seconds) I have over ...
3
votes
2answers
188 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
0
votes
1answer
39 views

Why does Googlebot-Mobile crawl desktop content?

Identical desktop and mobile URLs, no subdomain or sub-folder: Desktop URL site-name.com <link rel="canonical" href="index.php?/..."> Mobile URL site-name.com <link rel="canonical" ...
1
vote
1answer
85 views

Does Google bot crawl dynamically created iframes? [duplicate]

The situation looks as followed: User includes Javascript snippet into his page. Javascript creates iFrame dynamically and appends it to page. iFrame has static content. Is Google bot smart enough ...
1
vote
1answer
29 views

Serving a non-responsive css to ie8 and lower

We are rewritting our website to a responsive design (and maybe RESS in the future). Analytics figures impose us to support some old browsers (ie7 and ie8). We have 2 CSS: site.css: full css with ...
1
vote
0answers
42 views

Does 'thin-content' content on some pages affect overall site SEO?

I have an UGC Q&A site. Due to it's nature is inevitable to see some 'thin' or low quality content in some posts. Will there be any sitewide impact in SEO for the other high quality pages? If yes ...
9
votes
2answers
303 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
1
vote
1answer
55 views

Session ID Creating 404 Errors

Problem: Website contains URLs with parameters Session ID gets inserted into URLs when Googlebot crawls page with parameter (sorting) options Session ID is created and then expires, creating a 404 ...
0
votes
1answer
49 views

IP Geolocation and Googlebot

For a site that displays events near a city based on the user's IP address, how should Googlebot (or other crawlers) be handled? The site has many different pages and most of them use location-based ...
2
votes
3answers
91 views

A/B Testing A Site Redesign Without Google Penalties

I am redesigning our job board website to be responsive. Rather than just replace the existing site with the responsive site, I want to A/B test the new responsive site alongside the existing site so ...
2
votes
2answers
130 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: ...
2
votes
1answer
69 views

English content on International page templates: how to mark it up with lang=“” attributes?

So, I have translated page templates to various languages so it's easier for users to pick their language for easier navigating and understanding the site. The pages have similar markup to this ...
1
vote
2answers
65 views

Bandwidth saturation due to Googlebot search for spam pages

I recently had to update the hosting plan of a WordPress site I run due to bandwidth excess, which was already quite high. I've installed a security plugin that keeps log of 404 errors. From here I ...
0
votes
0answers
8 views

Incorrect Titles, Descriptions and Redirects in Google [duplicate]

I'm at a loss for why Google is indexing my site incorrectly. I just uploaded a new version of the site last night and "fetched as google" last night as well. Also uploaded sitemap. ...
1
vote
1answer
31 views

Is it harmful to block google while I continue to build content?

I have been blocking all IPs to my website (except my own) while it is in development. But now I am at a point, where I want other people to be able to see it, but I still don't want googlebot on it ...
2
votes
1answer
36 views

deindexing a whole domain with robots.txt created problems in google webmaster tools

In my robots.txt for my domain that just hosts images, I used the following lines: user-agent: * noindex: / I thought I was doing search engines a wonderful favor by not having them crawl through ...
0
votes
2answers
142 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
3
votes
1answer
74 views

Should I remove Google indexed pages before requesting a review after thin content manual action

Following on from my previous question - NOINDEX large number of pages to fix Google thin content manual action? Firstly, If I noindex all pages causing the manual action, is that enough for me to ...
6
votes
1answer
115 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
1
vote
0answers
44 views

Search engine ranking of paginated categories, subcategories and tag pages

Suppose we have an E-commerce domain with the following structure: domain/items - contains all items. domain/items/category - contains a narrower spectrum of results than above. ...
1
vote
1answer
59 views

SEO multiple ecommerce website running off the same codebase

I was wondering the about the effect a duplicate/similar codebase of products would have on a sites SEO across 190 domains. We have a dealer group with 190 for a certain class of products e.g. ...
1
vote
2answers
108 views

Issuu - PDF - Duplicated Content

I often write newsletters and publish them on my own site as PDF's. To potentially drive more traffic to my site, I was thinking about publishing the same newsletter files through sites like Scribd ...
0
votes
0answers
44 views

Odd requests from AdsBot-Google which re-include the domain name

It appears that since we started an AdWords campaign, our servers get tons of bogus requests from AdsBot-Google that for some reason include our own domain name in them. Example, if our website has ...
0
votes
2answers
101 views

Setting a preferred domain in Google webmaster tools

I have a site like "example.com". Questions 1: I can access to my site through "example.com" and www.example.com in web browser. Both non www and www version redirects to example.com. So, is it ...
1
vote
1answer
143 views

How to serve for International Googlebots if we redirect users to subdomains for language matching?

So, we have a site and a few sub-domains with content in different languages and we're using Hreflang and canonical implementations for SEO. We do detect the language that users use in their browsers ...
0
votes
1answer
43 views

Submission of nested sitemaps and nesting levels

I'm considering nested site maps for a site with many small files. I'm thinking of nesting sitemaps going 3 levels deep into the directories. Question 1: When you submit nested sitemaps to Google: ...
0
votes
0answers
198 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
1
vote
2answers
47 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
1
vote
3answers
44 views

Strategy for mobile redirection and using alternative and canonical links

I had a separate mobile site uploaded so as to make Google happy. My big problem is that I had a desktop page that was once showing in the top 3 results on Google, and now it and its mobile ...
1
vote
1answer
164 views

Fetch as Google fails in Search Console (Webmaster Tools)

Most of the time, the Fetch as Google succeeds only partially. There's always some resources that cannot be retrieved. It says the URLs are temporarily unreachable. I thought it was something to do ...
2
votes
1answer
95 views

Is google penalizing SEO of websites with non device-width viewports?

These articles make it look like they are: https://support.google.com/webmasters/answer/6101188?hl=en https://developers.google.com/web/fundamentals/layouts/rwd-fundamentals/set-the-viewport This ...
1
vote
3answers
323 views

Googlebot couldn't get all resources for this page

When I fetch and render it is showing partial. It looks fine. The same way Google sees it is the same way visitors see it. The reason for this is that I've added BuySellaAd codes in a widget onto my ...
2
votes
1answer
77 views

When will Google report correct results with its fetch tool?

I visited Google Webmaster Tools(Google Search Console) today after resubmitting sitemaps for my website, and while Google is happy to index more HTML pages of my site, it however has not indexed a ...
4
votes
0answers
553 views

How Googlebot discovers new website? [duplicate]

I have purchased a domain and uploaded with HTML Page and When I search in Google it shows my website. Now my question is How Google will discover my new website? I have not used ...
0
votes
0answers
103 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
0
votes
1answer
110 views

Why does Google Cache break my iframes?

In HTML5, iframe comes as a replacement for frameset. However, when I load my iframe-based page in SeaMonkey through Google Cache, the page appears empty, other than the standard Google Cache header ...