Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
2answers
49 views

Crawl Budget economy: what to do with URLs with tracking parameters?

So, a site has inbound links to a lot of important pages and these links have URL parameters. These URL parameters are basically used just for tracking purposes, similar to the way Amazon links to its ...
1
vote
1answer
42 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
29 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
-2
votes
0answers
19 views

SEO - Indexing Search Results and Pagination [on hold]

My website has undergone many changes over the past 10 years, as I guess most have, rankings have dropped and over the past few months I have been working on the site and CMS a lot to improve it, but ...
0
votes
0answers
11 views

SEO - Canonical Tags and, 301 and Product Variations

My website has undergone many changes over the past 10 years, as I guess most have, rankings have dropped and over the past few months I have been working on the site and CMS a lot to improve it, but ...
2
votes
1answer
56 views

Can you use Fetch as Google to get a page crawled and indexed on a domain you don't control?

I love Google Webmaster Fetch as Google to get the page crawled for freshness. Is there a way to do something similar on a domain I don't control? I can't add the specific domain in Google Webmaster ...
4
votes
3answers
870 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
4
votes
3answers
1k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
4
votes
4answers
93 views

Indexing too many pages good or bad?

We have a football website, in the website each football match has a separate page (for match details), and we decided to make unique title for each match page. Now I find many of these matches ...
3
votes
2answers
163 views

Does Google index AJAX websites after running the JavaScript?

As I understand Google depreciated its ajax crawling scheme they came up with back in 2009. I run single page app powered by Anguar frontend with push state navigation and my question is around ...
0
votes
0answers
11 views

It's 2016. How much JavaScript does Google understand? [duplicate]

Google deprecated its AJAX crawling scheme which seems to suggest that it can now index JavaScript applications. People now start testing the limits of what the bot can do with JavaScript, with ...
3
votes
1answer
37 views

Can I “cloak” for crawlers and not use session id parameters and redirects to check for cookies?

I track my users using GET parameters to be able to do it even if they don't allow the use of cookies. And to do that, I need to do 2 redirections the first time the user hits my page. I am having ...
3
votes
1answer
58 views

How to redirect googlebot with mobile user agent to the mobile version, if users are redirected with media query

The setup is absolutely so, as Google recommends: www.example.com has a media query for narrower display sizes and redirects users to m.example.com, if display becomes narrower, as setting. Both ...
2
votes
2answers
36 views

When would Google crawler give up crawling a URL with 5xx error

How many times will Google crawler attempt to recrawl a URL that has a 5xx error occurring intermittently? When would it give up and drop the URL from the Index?
2
votes
2answers
150 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
0
votes
1answer
39 views

Does Google Indexes content generated by JavaScript “On Scroll Load More”?

I'm implementing a "on scroll load more" page on my website and I'm wondering if google, now that crawls and indexes JavaScript, will see my content, does it?
0
votes
1answer
258 views

Does Googlebot see dynamically loaded content? [closed]

Our modelling agency website uses the Photolux Wordpress theme. Our front page uses the same masonry template. The only difference is that we've modified the masonry elements to link to a post instead ...
1
vote
0answers
7 views

Everything looks fine in the Search Console, but Google is not really indexing my site [duplicate]

I hope this question is not too generic. Some weeks ago, I set up a new Q&A site for my collaboration. I have added it to my search console dash board and I have submitted the sitemap. Google ...
1
vote
1answer
17 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
1
vote
2answers
643 views

Google Adsbot suddenly crawling odd pages

I have an error checker on my password reset page which tells me everytime someone comes to reset their password without a valid token. I have noticed within the last couple of days that the AdsBot-...
2
votes
0answers
22 views

Can a span within an H1 harm bot attempts to identify the text? [duplicate]

I am wondering if this structure can confuse search bots and prevent them from identifying the contents of the H1. For example, in the code below, can the bot identify "Main Text" as the H1 content? ...
0
votes
0answers
16 views

Google only indexing 2% of my sitemap on a 2 year old site [duplicate]

Looking at the Crawl stats on Google Webmaster Tools has made me very confused as to why Google is only indexing about 2-3% of the URL's in my sitemap. My sitemap contains 5288 URL's, but only 134 ...
0
votes
0answers
15 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
0
votes
2answers
54 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
0
votes
0answers
16 views

Is it legal to have an uncensored chat on a website

I am from Canada and have a website hosted on an American web server. Are their any laws for websites forcing any sort of censorship for user generated content (Chat containing Text/links only)? As ...
2
votes
1answer
150 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
0
votes
0answers
14 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
5
votes
1answer
46 views

301 and no-index tag don't work

Almost 3 months ago I've put no-index tag to about 3,000 pages on my website and 301 redirect (which is permanent redirect) for almost 15,000 pages. (The all site is about 50,000 pages) And yet as ...
1
vote
2answers
131 views

Issuu - PDF - Duplicated Content

I often write newsletters and publish them on my own site as PDF's. To potentially drive more traffic to my site, I was thinking about publishing the same newsletter files through sites like Scribd ...
1
vote
1answer
32 views

Using cookies on a site to redirect based on location - Googlebot crawling

I've recently began working on a Wordpress site that uses cookies to redirect a user to the appropriate regional page based on their location with geolocation. One thing I've noticed is that this ...
2
votes
3answers
126 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
7
votes
4answers
3k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
7
votes
3answers
961 views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
3
votes
1answer
185 views

Will Google index a React JS SPA if I can't add a rewrite rule?

The React SPA is set up to dynamically pull content from a server based on the URL. There is an index.html file sitting in a directory that routes to the correct content from the page requests. The ...
2
votes
1answer
85 views

How come Google can properly fetch and render some sites using a SPA but not others?

I have 2 websites, both use a single-page app framework. One uses React, and one uses Angular. Neither of them have any server-side rendering going on. When I run the "fetch and render as Google" ...
3
votes
2answers
48 views

Googlebot is still attempting to crawl old content

I have content that was deleted several years ago and from time to time Googlebot still attempts to access those pages, filling up my logs with lots of 404, making the 'real' problems harder to find ...
0
votes
1answer
57 views

We want to change the URLs of an entire section of the website, what all can go wrong with SEO?

We have the following URL sections: http://www.movie-website.com/movies/movie-name/trailers And we want to change it to: http://www.movie-website.com/trailers/movie-name/ Of course we are looking ...
2
votes
0answers
47 views

Google Analytics showing (not set) when viewing by Page Title

I'm trying to figure out why there is a row that shows a page title as (not set) in the Site Content > All Pages > Page Title table. I've read that this issue can be caused by improperly set ...
2
votes
2answers
77 views

Google displays search results even after removing URLs in Search Console

I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" ...
0
votes
0answers
31 views

Submit and Index page/url changes in Google Search Console

I made some changes to a page and it's url, and have configured my web server to redirect from the old to the new. After fetching the old and new url's in the search console the results appear as ...
24
votes
1answer
2k views

Why does my IP address become Google's IP address when using Chrome on the mobile?

I am tracking every member's geolocation (using db-ip) and everything is fine except mobile phones with the Chrome browser. They always give me the result US Mountain View California ip:66.249.xxx....
2
votes
2answers
45 views

Google crawler accessing non existing pages

I am looking Search Console and found Crawl errors for following URLs on my site: example.com/mobile example.com/m I never mentioned these pages anywhere but why Google is accessing it and what ...
4
votes
5answers
1k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
3answers
83 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
0
votes
0answers
14 views

Domain expired and was down for about 30 min, will it effect Google ranking? [duplicate]

I've discovered my domain expired, apparently about 20 min after it expired. it took few min to renew it and few min until the name servers reverted correctly. Is there a chance that this downtime ...
1
vote
0answers
66 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
3
votes
1answer
45 views

Structured Data Tool doesn't see JavaScript rendered content

Some structured data (namely price) of my page is rendered by JavaScript. According to fetching results in Webmaster tools Googlebot sees and renders it perfectly. However the structured data tool ...
3
votes
1answer
230 views

Changing URLs in Google's Index

I recently submitted a sitemap.xml in Google's Webmaster Tools and some of the pages already got indexed. The problem now is, that I cleaned my URLs with MOD_REWRITE in the .htaccess file afterwards ...
2
votes
3answers
101 views

A/B Testing A Site Redesign Without Google Penalties

I am redesigning our job board website to be responsive. Rather than just replace the existing site with the responsive site, I want to A/B test the new responsive site alongside the existing site so ...
1
vote
3answers
384 views

Googlebot couldn't get all resources for this page

When I fetch and render it is showing partial. It looks fine. The same way Google sees it is the same way visitors see it. The reason for this is that I've added BuySellaAd codes in a widget onto my ...