Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
1answer
31 views

Googlebot spamming?

We run a sailing website and offer measurement forms for sails and what not -- recently we've been getting hit with hundreds of form submissions with random letters and names in the blanks and it's ...
0
votes
0answers
6 views

Whether adding more content in single page will create SEO problems? [duplicate]

I am planned to start a quiz site for C++ interview questions. Instead of taking user to different pages i decided to keep all the questions is same page. When user click the button of the ...
0
votes
1answer
30 views

Google-Image bot not respecting robots.txt

I have the following in my robots.txt file: # Block Googlebot-Image from cache User-agent: Googlebot-Image Disallow: /media/catalog/product/cache/ # Crawlers Setup User-agent: * # Directories ...
2
votes
1answer
71 views

Can you use Fetch as Google to get a page crawled and indexed on a domain you don't control?

I love Google Webmaster Fetch as Google to get the page crawled for freshness. Is there a way to do something similar on a domain I don't control? I can't add the specific domain in Google Webmaster ...
3
votes
2answers
185 views

Does Google index AJAX websites after running the JavaScript?

As I understand Google depreciated its ajax crawling scheme they came up with back in 2009. I run single page app powered by Anguar frontend with push state navigation and my question is around ...
0
votes
1answer
24 views

Does Web crawler simulate user behavior?

Structuring the web pages' content or schema properly is important for SEO. I would like to know if popular web crawlers or bots (Google bot, Bingbot etc...) are able to simulate user behavior and ...
3
votes
1answer
115 views

Is google penalizing SEO of websites with non device-width viewports?

These articles make it look like they are: https://support.google.com/webmasters/answer/6101188?hl=en https://developers.google.com/web/fundamentals/layouts/rwd-fundamentals/set-the-viewport This ...
3
votes
2answers
32 views

Why is Googlebot calling an AJAX method with GET rather than POST?

I have an AJAX call on my site that does a POST and returns data. Googlebot is crawling these URLs but it is attempting to use a GET method rather than a POST. This is causing hundreds of 404 error ...
1
vote
1answer
411 views

Does a jQuery “rel=nofollow” even work?

It's a fairly common practice for webmasters to rel=nofollow external links, and I frequently see it done with jQuery. The script typically looks something like this (off the cuff here): jQuery(...
4
votes
1answer
77 views

How to redirect googlebot with mobile user agent to the mobile version, if users are redirected with media query

The setup is absolutely so, as Google recommends: www.example.com has a media query for narrower display sizes and redirects users to m.example.com, if display becomes narrower, as setting. Both ...
1
vote
1answer
41 views

How disqus comments (inside cross domain iframe) are indexed by google?

According to what I have read is that google doesn't index content from any cross domain iframe since it's the content of another website. So how does Disqus do it?
3
votes
2answers
176 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
0
votes
0answers
9 views

Do I need multiple properties in Google Search Console with all of my domain variations? [duplicate]

All non https links in my website are redirected via nginx to https. http://kida.al/ and www.kida.al are redirected to https://kida.al/ I have three properties in Search Console. kida.al www.kida....
1
vote
1answer
53 views

Why is hidden content not always indexed?

I've read several blogs discussing the 'hidden content' issue. This article experimented with several options. It concludes that Google DOES index hidden content. The issues is also discussed here. ...
0
votes
1answer
28 views

Question About Traffic From Google And GEO Location OF Website

I want to know lets say a person who lives in South Wimbledon, UK, runs a website and then outsource the website work (like writing articles and posting and other site management work etc..) to a ...
0
votes
1answer
133 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. http://www.example.com/segment1/segment2&sa=U&ved=...
1
vote
2answers
32 views

At what resolution does Google Bot Web Crawlers consider desktop?

I'm aware that google evaluates how a site looks on mobile. At what resolution does google evaluate desktop sites? My site uses a responsive layout up until 1200px width, at which point it shows and ...
4
votes
3answers
78 views

Googlebot sends several hundred requests in a minute

I have a website which has several million page views per month. Starting several days ago, a bot named Googlebot sends requests to the site very frequently. It sends several hundred requests to the ...
1
vote
1answer
27 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
0
votes
1answer
30 views

Is it legal to have an uncensored chat on a website

I am from Canada and have a website hosted on an American web server. Are their any laws for websites forcing any sort of censorship for user generated content (Chat containing Text/links only)? As ...
2
votes
1answer
20 views

SEO - Canonical Tags and, 301 and Product Variations

My website has undergone many changes over the past 10 years, as I guess most have, rankings have dropped and over the past few months I have been working on the site and CMS a lot to improve it, but ...
1
vote
1answer
71 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
1answer
80 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
1
vote
1answer
113 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: www.example.com?myquery&...
1
vote
1answer
121 views

Googlebot follows rel=“nofollow” links

I would like to hide all pages of our website from being read or indexed by Googlebot, except several thousand of pages which were selected for indexing. The site was launched in April 2015. The ...
1
vote
1answer
37 views

Does fetch and render + submit to index interrupt Google's crawl period?

If you resubmit your site to the index in Webmaster Tools, will pending crawls get cancelled? I ask because a client's site has been up for 72 hours now and has not been crawled. I submitted to index ...
3
votes
1answer
282 views

Will Google index a React JS SPA if I can't add a rewrite rule?

The React SPA is set up to dynamically pull content from a server based on the URL. There is an index.html file sitting in a directory that routes to the correct content from the page requests. The ...
1
vote
2answers
51 views

Crawl Budget economy: what to do with URLs with tracking parameters?

So, a site has inbound links to a lot of important pages and these links have URL parameters. These URL parameters are basically used just for tracking purposes, similar to the way Amazon links to its ...
1
vote
1answer
46 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
32 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
4
votes
3answers
930 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
4
votes
3answers
1k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
4
votes
4answers
100 views

Indexing too many pages good or bad?

We have a football website, in the website each football match has a separate page (for match details), and we decided to make unique title for each match page. Now I find many of these matches ...
0
votes
0answers
13 views

It's 2016. How much JavaScript does Google understand? [duplicate]

Google deprecated its AJAX crawling scheme which seems to suggest that it can now index JavaScript applications. People now start testing the limits of what the bot can do with JavaScript, with ...
3
votes
1answer
43 views

Can I “cloak” for crawlers and not use session id parameters and redirects to check for cookies?

I track my users using GET parameters to be able to do it even if they don't allow the use of cookies. And to do that, I need to do 2 redirections the first time the user hits my page. I am having ...
2
votes
2answers
41 views

When would Google crawler give up crawling a URL with 5xx error

How many times will Google crawler attempt to recrawl a URL that has a 5xx error occurring intermittently? When would it give up and drop the URL from the Index?
0
votes
1answer
55 views

Does Google Indexes content generated by JavaScript “On Scroll Load More”?

I'm implementing a "on scroll load more" page on my website and I'm wondering if google, now that crawls and indexes JavaScript, will see my content, does it?
0
votes
1answer
269 views

Does Googlebot see dynamically loaded content? [closed]

Our modelling agency website uses the Photolux Wordpress theme. Our front page uses the same masonry template. The only difference is that we've modified the masonry elements to link to a post instead ...
1
vote
0answers
8 views

Everything looks fine in the Search Console, but Google is not really indexing my site [duplicate]

I hope this question is not too generic. Some weeks ago, I set up a new Q&A site for my collaboration. I have added it to my search console dash board and I have submitted the sitemap. Google ...
1
vote
1answer
18 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
1
vote
2answers
681 views

Google Adsbot suddenly crawling odd pages

I have an error checker on my password reset page which tells me everytime someone comes to reset their password without a valid token. I have noticed within the last couple of days that the AdsBot-...
2
votes
0answers
22 views

Can a span within an H1 harm bot attempts to identify the text? [duplicate]

I am wondering if this structure can confuse search bots and prevent them from identifying the contents of the H1. For example, in the code below, can the bot identify "Main Text" as the H1 content? ...
0
votes
0answers
16 views

Google only indexing 2% of my sitemap on a 2 year old site [duplicate]

Looking at the Crawl stats on Google Webmaster Tools has made me very confused as to why Google is only indexing about 2-3% of the URL's in my sitemap. My sitemap contains 5288 URL's, but only 134 ...
0
votes
0answers
21 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
0
votes
2answers
66 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
2
votes
1answer
167 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
5
votes
1answer
50 views

301 and no-index tag don't work

Almost 3 months ago I've put no-index tag to about 3,000 pages on my website and 301 redirect (which is permanent redirect) for almost 15,000 pages. (The all site is about 50,000 pages) And yet as ...
1
vote
2answers
135 views

Issuu - PDF - Duplicated Content

I often write newsletters and publish them on my own site as PDF's. To potentially drive more traffic to my site, I was thinking about publishing the same newsletter files through sites like Scribd ...
1
vote
1answer
42 views

Using cookies on a site to redirect based on location - Googlebot crawling

I've recently began working on a Wordpress site that uses cookies to redirect a user to the appropriate regional page based on their location with geolocation. One thing I've noticed is that this ...
2
votes
3answers
134 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?