Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

4
votes
3answers
837 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
0
votes
0answers
11 views

It's 2016. How much JavaScript does Google understand? [duplicate]

Google deprecated its AJAX crawling scheme which seems to suggest that it can now index JavaScript applications. People now start testing the limits of what the bot can do with JavaScript, with ...
3
votes
1answer
35 views

Can I “cloak” for crawlers and not use session id parameters and redirects to check for cookies?

I track my users using GET parameters to be able to do it even if they don't allow the use of cookies. And to do that, I need to do 2 redirections the first time the user hits my page. I am having ...
2
votes
2answers
35 views

When would Google crawler give up crawling a URL with 5xx error

How many times will Google crawler attempt to recrawl a URL that has a 5xx error occurring intermittently? When would it give up and drop the URL from the Index?
0
votes
1answer
39 views

Google is unable to crawl my redirected URL

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
0
votes
0answers
28 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
4
votes
4answers
90 views

Indexing too many pages good or bad?

We have a football website, in the website each football match has a separate page (for match details), and we decided to make unique title for each match page. Now I find many of these matches ...
1
vote
0answers
7 views

Everything looks fine in the Search Console, but Google is not really indexing my site [duplicate]

I hope this question is not too generic. Some weeks ago, I set up a new Q&A site for my collaboration. I have added it to my search console dash board and I have submitted the sitemap. Google ...
0
votes
1answer
39 views

Does Google Indexes content generated by JavaScript “On Scroll Load More”?

I'm implementing a "on scroll load more" page on my website and I'm wondering if google, now that crawls and indexes JavaScript, will see my content, does it?
1
vote
1answer
17 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
2
votes
0answers
22 views

Can a span within an H1 harm bot attempts to identify the text? [duplicate]

I am wondering if this structure can confuse search bots and prevent them from identifying the contents of the H1. For example, in the code below, can the bot identify "Main Text" as the H1 content? ...
0
votes
0answers
16 views

Google only indexing 2% of my sitemap on a 2 year old site [duplicate]

Looking at the Crawl stats on Google Webmaster Tools has made me very confused as to why Google is only indexing about 2-3% of the URL's in my sitemap. My sitemap contains 5288 URL's, but only 134 ...
0
votes
2answers
52 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
0
votes
0answers
16 views

Is it legal to have an uncensored chat on a website

I am from Canada and have a website hosted on an American web server. Are their any laws for websites forcing any sort of censorship for user generated content (Chat containing Text/links only)? As ...
0
votes
0answers
14 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
0
votes
0answers
14 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
5
votes
1answer
46 views

301 and no-index tag don't work

Almost 3 months ago I've put no-index tag to about 3,000 pages on my website and 301 redirect (which is permanent redirect) for almost 15,000 pages. (The all site is about 50,000 pages) And yet as ...
1
vote
1answer
30 views

Using cookies on a site to redirect based on location - Googlebot crawling

I've recently began working on a Wordpress site that uses cookies to redirect a user to the appropriate regional page based on their location with geolocation. One thing I've noticed is that this ...
3
votes
1answer
57 views

How to redirect googlebot with mobile user agent to the mobile version, if users are redirected with media query

The setup is absolutely so, as Google recommends: www.example.com has a media query for narrower display sizes and redirects users to m.example.com, if display becomes narrower, as setting. Both ...
7
votes
3answers
961 views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
2
votes
1answer
82 views

How come Google can properly fetch and render some sites using a SPA but not others?

I have 2 websites, both use a single-page app framework. One uses React, and one uses Angular. Neither of them have any server-side rendering going on. When I run the "fetch and render as Google" ...
3
votes
1answer
172 views

Will Google index a React JS SPA if I can't add a rewrite rule?

The React SPA is set up to dynamically pull content from a server based on the URL. There is an index.html file sitting in a directory that routes to the correct content from the page requests. The ...
3
votes
2answers
48 views

Googlebot is still attempting to crawl old content

I have content that was deleted several years ago and from time to time Googlebot still attempts to access those pages, filling up my logs with lots of 404, making the 'real' problems harder to find ...
2
votes
0answers
46 views

Google Analytics showing (not set) when viewing by Page Title

I'm trying to figure out why there is a row that shows a page title as (not set) in the Site Content > All Pages > Page Title table. I've read that this issue can be caused by improperly set ...
2
votes
2answers
75 views

Google displays search results even after removing URLs in Search Console

I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" ...
0
votes
0answers
29 views

Submit and Index page/url changes in Google Search Console

I made some changes to a page and it's url, and have configured my web server to redirect from the old to the new. After fetching the old and new url's in the search console the results appear as ...
24
votes
1answer
2k views

Why does my IP address become Google's IP address when using Chrome on the mobile?

I am tracking every member's geolocation (using db-ip) and everything is fine except mobile phones with the Chrome browser. They always give me the result US Mountain View California ip:66.249.xxx....
3
votes
3answers
82 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
0
votes
0answers
14 views

Domain expired and was down for about 30 min, will it effect Google ranking? [duplicate]

I've discovered my domain expired, apparently about 20 min after it expired. it took few min to renew it and few min until the name servers reverted correctly. Is there a chance that this downtime ...
1
vote
0answers
64 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
3
votes
1answer
230 views

Changing URLs in Google's Index

I recently submitted a sitemap.xml in Google's Webmaster Tools and some of the pages already got indexed. The problem now is, that I cleaned my URLs with MOD_REWRITE in the .htaccess file afterwards ...
2
votes
2answers
44 views

Google crawler accessing non existing pages

I am looking Search Console and found Crawl errors for following URLs on my site: example.com/mobile example.com/m I never mentioned these pages anywhere but why Google is accessing it and what ...
2
votes
1answer
53 views

Forcing google to rescan sitemaps instantly

When URLs no longer exist on my site, I give a page and issue an HTTP 410 status (the GONE error). I created a script that allows my co-administrator to add and remove pictures from the site and ...
1
vote
1answer
66 views

Why is this mobile/image Googlebot crawl rate so fast? (10-30 req per sec)

This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone Build/...
4
votes
1answer
67 views

Does Googlebot trigger onclick events and index the resulting content?

I've found many topics about Googlebot following links and executing ajax queries but none that tell me whether onclick events would get triggered and results crawled. Let's say that an image is ...
3
votes
1answer
44 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
3
votes
1answer
45 views

Structured Data Tool doesn't see JavaScript rendered content

Some structured data (namely price) of my page is rendered by JavaScript. According to fetching results in Webmaster tools Googlebot sees and renders it perfectly. However the structured data tool ...
1
vote
0answers
53 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
8
votes
5answers
1k views

Is it worthwhile to pay for boosting SEO for a new wesite?

I have come across a couple of websites today that claim to boost your website index in a matter of days. There are 1000s of reviews and all of them seem quite positive showing off results and stuff. ...
0
votes
1answer
57 views

We want to change the URLs of an entire section of the website, what all can go wrong with SEO?

We have the following URL sections: http://www.movie-website.com/movies/movie-name/trailers And we want to change it to: http://www.movie-website.com/trailers/movie-name/ Of course we are looking ...
1
vote
1answer
62 views

Google Master Tool: How allow external links in robots.txt

I have a problem with google rendering. Googlebot couldn't get all resources for this page. Because it's blocked by its own google robots.txt file. Here's an example of an external link. https://www....
3
votes
2answers
73 views

Increase in 5XX errors and 404 errors on GWT, are we under attack?

A lot of URLs are suddenly giving us 404s and 500s. On closer inspection, these URLs do not exist on our website and are being linked from random sources. As more URLs were inspected on GWT, a ...
3
votes
1answer
32 views

Google blocks map API, GWT shows severity high, will this cause SEO problems?

Google blocked map API, GWT showed severity high. What is the effect on SEO?
1
vote
1answer
27 views

What are the possible reasons for a sudden decrease in the number of images being indexed in my sitemap?

A number of sitemaps contain URLs, and a number of those URLs contain image URLs with captions, titles, etc. Suddenly over the past five days, there has been a drop in the number of images indexed by ...
0
votes
1answer
23 views

What affect, if any, would buying domain names with high volume search terms have on SEO?

Assume the following: I have a website: www.example.com. I want to rank www.example.com/searc-term-1 on the first page of Google. Would it help the SEO of www.example.com/searc-term-1 if I bought ...
1
vote
2answers
47 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
2answers
33 views

Are there any meta tags I can add to my website to make it more discoverable by search engines?

I am quite a beginner at website design, and in fact so much of an amateur that I don't like paying for hosting/higher search engine results. Are there any meta's that will simply make my website a ...
1
vote
0answers
59 views

Does Google still crawl pages with dynamically changing content like news feeds?

Some websites like the following have a news feed on their homepage that provides the majority of content, which is an increasingly popular feature: https://www.producthunt.com/ https://www.tripoto....
2
votes
3answers
126 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
1
vote
2answers
50 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?