Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
1answer
27 views

Googlebot spamming?

We run a sailing website and offer measurement forms for sails and what not -- recently we've been getting hit with hundreds of form submissions with random letters and names in the blanks and it's ...
0
votes
0answers
6 views

Whether adding more content in single page will create SEO problems? [duplicate]

I am planned to start a quiz site for C++ interview questions. Instead of taking user to different pages i decided to keep all the questions is same page. When user click the button of the ...
0
votes
1answer
30 views

Google-Image bot not respecting robots.txt

I have the following in my robots.txt file: # Block Googlebot-Image from cache User-agent: Googlebot-Image Disallow: /media/catalog/product/cache/ # Crawlers Setup User-agent: * # Directories ...
0
votes
1answer
24 views

Does Web crawler simulate user behavior?

Structuring the web pages' content or schema properly is important for SEO. I would like to know if popular web crawlers or bots (Google bot, Bingbot etc...) are able to simulate user behavior and ...
3
votes
2answers
30 views

Proper Robots.txt Block URL

I want to block the url like: /services/online.html?dir=asc&limit=12&order=price but I want to allow: /services/online.html I have added this on the Robots.txt: Allow: /services/online....
3
votes
2answers
32 views

Why is Googlebot calling an AJAX method with GET rather than POST?

I have an AJAX call on my site that does a POST and returns data. Googlebot is crawling these URLs but it is attempting to use a GET method rather than a POST. This is causing hundreds of 404 error ...
1
vote
1answer
41 views

How disqus comments (inside cross domain iframe) are indexed by google?

According to what I have read is that google doesn't index content from any cross domain iframe since it's the content of another website. So how does Disqus do it?
0
votes
0answers
9 views

Do I need multiple properties in Google Search Console with all of my domain variations? [duplicate]

All non https links in my website are redirected via nginx to https. http://kida.al/ and www.kida.al are redirected to https://kida.al/ I have three properties in Search Console. kida.al www.kida....
1
vote
2answers
32 views

At what resolution does Google Bot Web Crawlers consider desktop?

I'm aware that google evaluates how a site looks on mobile. At what resolution does google evaluate desktop sites? My site uses a responsive layout up until 1200px width, at which point it shows and ...
4
votes
3answers
78 views

Googlebot sends several hundred requests in a minute

I have a website which has several million page views per month. Starting several days ago, a bot named Googlebot sends requests to the site very frequently. It sends several hundred requests to the ...
2
votes
1answer
19 views

SEO - Canonical Tags and, 301 and Product Variations

My website has undergone many changes over the past 10 years, as I guess most have, rankings have dropped and over the past few months I have been working on the site and CMS a lot to improve it, but ...
4
votes
3answers
929 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
0
votes
0answers
12 views

It's 2016. How much JavaScript does Google understand? [duplicate]

Google deprecated its AJAX crawling scheme which seems to suggest that it can now index JavaScript applications. People now start testing the limits of what the bot can do with JavaScript, with ...
3
votes
1answer
43 views

Can I “cloak” for crawlers and not use session id parameters and redirects to check for cookies?

I track my users using GET parameters to be able to do it even if they don't allow the use of cookies. And to do that, I need to do 2 redirections the first time the user hits my page. I am having ...
2
votes
2answers
41 views

When would Google crawler give up crawling a URL with 5xx error

How many times will Google crawler attempt to recrawl a URL that has a 5xx error occurring intermittently? When would it give up and drop the URL from the Index?
1
vote
1answer
46 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
32 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
4
votes
4answers
100 views

Indexing too many pages good or bad?

We have a football website, in the website each football match has a separate page (for match details), and we decided to make unique title for each match page. Now I find many of these matches ...
1
vote
0answers
8 views

Everything looks fine in the Search Console, but Google is not really indexing my site [duplicate]

I hope this question is not too generic. Some weeks ago, I set up a new Q&A site for my collaboration. I have added it to my search console dash board and I have submitted the sitemap. Google ...
0
votes
1answer
55 views

Does Google Indexes content generated by JavaScript “On Scroll Load More”?

I'm implementing a "on scroll load more" page on my website and I'm wondering if google, now that crawls and indexes JavaScript, will see my content, does it?
1
vote
1answer
18 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
2
votes
0answers
22 views

Can a span within an H1 harm bot attempts to identify the text? [duplicate]

I am wondering if this structure can confuse search bots and prevent them from identifying the contents of the H1. For example, in the code below, can the bot identify "Main Text" as the H1 content? ...
0
votes
0answers
16 views

Google only indexing 2% of my sitemap on a 2 year old site [duplicate]

Looking at the Crawl stats on Google Webmaster Tools has made me very confused as to why Google is only indexing about 2-3% of the URL's in my sitemap. My sitemap contains 5288 URL's, but only 134 ...
0
votes
2answers
66 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
0
votes
1answer
30 views

Is it legal to have an uncensored chat on a website

I am from Canada and have a website hosted on an American web server. Are their any laws for websites forcing any sort of censorship for user generated content (Chat containing Text/links only)? As ...
0
votes
0answers
21 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
1
vote
1answer
27 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
5
votes
1answer
50 views

301 and no-index tag don't work

Almost 3 months ago I've put no-index tag to about 3,000 pages on my website and 301 redirect (which is permanent redirect) for almost 15,000 pages. (The all site is about 50,000 pages) And yet as ...
1
vote
1answer
40 views

Using cookies on a site to redirect based on location - Googlebot crawling

I've recently began working on a Wordpress site that uses cookies to redirect a user to the appropriate regional page based on their location with geolocation. One thing I've noticed is that this ...
4
votes
1answer
77 views

How to redirect googlebot with mobile user agent to the mobile version, if users are redirected with media query

The setup is absolutely so, as Google recommends: www.example.com has a media query for narrower display sizes and redirects users to m.example.com, if display becomes narrower, as setting. Both ...
7
votes
3answers
962 views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
2
votes
1answer
111 views

How come Google can properly fetch and render some sites using a SPA but not others?

I have 2 websites, both use a single-page app framework. One uses React, and one uses Angular. Neither of them have any server-side rendering going on. When I run the "fetch and render as Google" ...
3
votes
1answer
281 views

Will Google index a React JS SPA if I can't add a rewrite rule?

The React SPA is set up to dynamically pull content from a server based on the URL. There is an index.html file sitting in a directory that routes to the correct content from the page requests. The ...
3
votes
2answers
54 views

Googlebot is still attempting to crawl old content

I have content that was deleted several years ago and from time to time Googlebot still attempts to access those pages, filling up my logs with lots of 404, making the 'real' problems harder to find ...
2
votes
0answers
67 views

Google Analytics showing (not set) when viewing by Page Title

I'm trying to figure out why there is a row that shows a page title as (not set) in the Site Content > All Pages > Page Title table. I've read that this issue can be caused by improperly set ...
2
votes
2answers
98 views

Google displays search results even after removing URLs in Search Console

I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" ...
0
votes
0answers
40 views

Submit and Index page/url changes in Google Search Console

I made some changes to a page and it's url, and have configured my web server to redirect from the old to the new. After fetching the old and new url's in the search console the results appear as ...
24
votes
1answer
2k views

Why does my IP address become Google's IP address when using Chrome on the mobile?

I am tracking every member's geolocation (using db-ip) and everything is fine except mobile phones with the Chrome browser. They always give me the result US Mountain View California ip:66.249.xxx....
3
votes
3answers
92 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
0
votes
0answers
14 views

Domain expired and was down for about 30 min, will it effect Google ranking? [duplicate]

I've discovered my domain expired, apparently about 20 min after it expired. it took few min to renew it and few min until the name servers reverted correctly. Is there a chance that this downtime ...
1
vote
0answers
72 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
3
votes
1answer
231 views

Changing URLs in Google's Index

I recently submitted a sitemap.xml in Google's Webmaster Tools and some of the pages already got indexed. The problem now is, that I cleaned my URLs with MOD_REWRITE in the .htaccess file afterwards ...
2
votes
2answers
53 views

Google crawler accessing non existing pages

I am looking Search Console and found Crawl errors for following URLs on my site: example.com/mobile example.com/m I never mentioned these pages anywhere but why Google is accessing it and what ...
2
votes
1answer
57 views

Forcing google to rescan sitemaps instantly

When URLs no longer exist on my site, I give a page and issue an HTTP 410 status (the GONE error). I created a script that allows my co-administrator to add and remove pictures from the site and ...
1
vote
1answer
70 views

Why is this mobile/image Googlebot crawl rate so fast? (10-30 req per sec)

This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone Build/...
4
votes
1answer
79 views

Does Googlebot trigger onclick events and index the resulting content?

I've found many topics about Googlebot following links and executing ajax queries but none that tell me whether onclick events would get triggered and results crawled. Let's say that an image is ...
3
votes
1answer
47 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
3
votes
1answer
57 views

Structured Data Tool doesn't see JavaScript rendered content

Some structured data (namely price) of my page is rendered by JavaScript. According to fetching results in Webmaster tools Googlebot sees and renders it perfectly. However the structured data tool ...
1
vote
0answers
55 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
8
votes
5answers
1k views

Is it worthwhile to pay for boosting SEO for a new wesite?

I have come across a couple of websites today that claim to boost your website index in a matter of days. There are 1000s of reviews and all of them seem quite positive showing off results and stuff. ...