Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

3
votes
2answers
162 views

Does Google index AJAX websites after running the JavaScript?

As I understand Google depreciated its ajax crawling scheme they came up with back in 2009. I run single page app powered by Anguar frontend with push state navigation and my question is around ...
2
votes
2answers
149 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
3
votes
1answer
57 views

How to redirect googlebot with mobile user agent to the mobile version, if users are redirected with media query

The setup is absolutely so, as Google recommends: www.example.com has a media query for narrower display sizes and redirects users to m.example.com, if display becomes narrower, as setting. Both ...
2
votes
1answer
56 views

Can you use Fetch as Google to get a page crawled and indexed on a domain you don't control?

I love Google Webmaster Fetch as Google to get the page crawled for freshness. Is there a way to do something similar on a domain I don't control? I can't add the specific domain in Google Webmaster ...
2
votes
1answer
148 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
0
votes
1answer
39 views

Google is unable to crawl my redirected URL

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
2
votes
0answers
46 views

Google Analytics showing (not set) when viewing by Page Title

I'm trying to figure out why there is a row that shows a page title as (not set) in the Site Content > All Pages > Page Title table. I've read that this issue can be caused by improperly set ...
2
votes
0answers
231 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
1
vote
0answers
64 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
1
vote
0answers
53 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
1
vote
0answers
59 views

Does Google still crawl pages with dynamically changing content like news feeds?

Some websites like the following have a news feed on their homepage that provides the majority of content, which is an increasingly popular feature: https://www.producthunt.com/ https://www.tripoto....
1
vote
0answers
32 views

Does fetch and render + submit to index interrupt Google's crawl period?

If you resubmit your site to the index in Webmaster Tools, will pending crawls get cancelled? I ask because a client's site has been up for 72 hours now and has not been crawled. I submitted to index ...
1
vote
0answers
45 views

Does 'thin-content' content on some pages affect overall site SEO?

I have an UGC Q&A site. Due to it's nature is inevitable to see some 'thin' or low quality content in some posts. Will there be any sitewide impact in SEO for the other high quality pages? If yes ...
1
vote
0answers
46 views

Search engine ranking of paginated categories, subcategories and tag pages

Suppose we have an E-commerce domain with the following structure: domain/items - contains all items. domain/items/category - contains a narrower spectrum of results than above. domain/items/...
1
vote
0answers
55 views

How post or page of facebook, twitter, forum or classified sites (HIGH PR) show instantally in google or other search engines

My question is about SEO. How post or page of facebook, twitter, forum or classified sites(HIGH PR) show instantally in google or other search engines. As i know google take few days for crawl. Then ...
1
vote
0answers
117 views

Googlebot follows rel=“nofollow” links

I would like to hide all pages of our website from being read or indexed by Googlebot, except several thousand of pages which were selected for indexing. The site was launched in April 2015. The ...
1
vote
0answers
141 views

google displays wrong language on multi language website because it takes cookies into consideration

I have a multiple language website that uses both subdomains and cookies to display the right language. Here is the logic on the website: de subdomain will display the German version fr subdomain ...
1
vote
0answers
413 views

Google Webmaster Tools can't access robots.txt and the home page is “temporarily unreachable”

My site http://goo.gl/xH2Ffy suddenly disappeared from Google search and Webmaster Tools says it can't access robots.txt. Fetch as Google says "temporarily unreachable" for robots.txt and even for the ...
1
vote
0answers
108 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: www.example.com?myquery&...
1
vote
0answers
92 views

I still get “Googlebot can't access your site” messages after removing a site from Google Webmaster Tools

I had a client who moved their site a while back to another developer. When they moved it they took down the site and put up a coming soon page ( which is currently what's there). Googlebot warned me ...
1
vote
0answers
113 views

Why does Google keep hitting only one MP3 file?

I have a bunch of Mp3 files on my site, and all if them are in the sitemap to allow search engines that log mp3 files to know about them. For some reason, every day or two, Google keeps hitting the ...
1
vote
0answers
75 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
1
vote
0answers
66 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
0answers
201 views

Site returning 404 header to Google, not sure why

A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by Google at all (which is the main issue but I ...
0
votes
0answers
28 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
0
votes
0answers
16 views

Is it legal to have an uncensored chat on a website

I am from Canada and have a website hosted on an American web server. Are their any laws for websites forcing any sort of censorship for user generated content (Chat containing Text/links only)? As ...
0
votes
0answers
14 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
0
votes
0answers
14 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
0
votes
0answers
29 views

Submit and Index page/url changes in Google Search Console

I made some changes to a page and it's url, and have configured my web server to redirect from the old to the new. After fetching the old and new url's in the search console the results appear as ...
0
votes
0answers
47 views

Why is hidden content not always indexed?

I've read several blogs discussing the 'hidden content' issue. This article experimented with several options. It concludes that Google DOES index hidden content. The issues is also discussed here. ...
0
votes
0answers
269 views

SEO on ui-router states of AngularJS SPA

I have a simple SPA created with AngularJS, you can see it running here: www.consami.com.br. I was using Prerender.io to load a pure HTML version for crawlers, however some pages were not indexed by ...
0
votes
0answers
42 views

Crawl Budget economy: what to do with URLs with parameters?

So, a site has inbound links to a lot of important pages and these links have URL parameters. These URL parameters are basically used just for tracking purposes, similar to the way Amazon links to its ...
0
votes
0answers
38 views

access.log requests from UA “Google favicon”…? What do these mean?

I have a bunch of these in my nginx log file, on a range of URLs: on 66.102.9.23 - - [12/Nov/2015:10:40:25 +0000] "GET /myadminfolder/page.html HTTP/1.1" 200 1247 1800 "-" "Google favicon" "...
0
votes
0answers
24 views

How should I render a widget with many links inside a blog so google don't penalize the blog?

We run a service (will not mention the URL, as this is not a pitch) and need advice on Google-viewing-links matters. The service is for bloggers and allows them to create a "linkup" inside their blog ...
0
votes
0answers
45 views

Odd requests from AdsBot-Google which re-include the domain name

It appears that since we started an AdWords campaign, our servers get tons of bogus requests from AdsBot-Google that for some reason include our own domain name in them. Example, if our website has ...
0
votes
0answers
240 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
0
votes
0answers
123 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. http://www.example.com/segment1/segment2&sa=U&ved=...
0
votes
0answers
27 views

Google Crawler constantly reports ~2-3% errors

For the last year and a half, I have a constant 2-3% error rate while crawling my web site. Types of errors are "Server error: No response". I've switched ISP, but errors persist. Debugging of ...
0
votes
0answers
24 views

Question About Traffic From Google And GEO Location OF Website

I want to know lets say a person who lives in South Wimbledon, UK, runs a website and then outsource the website work (like writing articles and posting and other site management work etc..) to a ...