Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

4
votes
1answer
229 views

How to hide a JavaScript popup for Googlebot?

I want to display a JavaScript popup when visitors access my website, however I believe Google will decrease my rankings when they detect a popup. So, is there any way to hide a JavaScript popup for ...
4
votes
2answers
397 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
4
votes
1answer
2k views

IIS Dynamic IP Restrictions Module and Googlebot

I consider using IIS Dynamic IP Restrictions Module in order to make it hard for DoS (Denial of Service) attacks. But I am not sure how that would work out for search engine bots like Googlebot. Any ...
4
votes
3answers
1k views

nofollow doesn't stop crawling?

In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" href="/link/share/linkedin/Dance+Combo+%28Jazz%2C+Tap%2C+...
4
votes
3answers
290 views

Why is the number of pages on my website indexed by Googlebot decreasing?

I am monitoring the number of pages indexed by Googlebot for one of my websites . I noticed that since last week, the number of pages has dropped by about 60 or so. Now to be fair, I have not added ...
4
votes
1answer
68 views

Does Googlebot trigger onclick events and index the resulting content?

I've found many topics about Googlebot following links and executing ajax queries but none that tell me whether onclick events would get triggered and results crawled. Let's say that an image is ...
4
votes
1answer
79 views

Does Google now factor the prominence of how a link is rendered into their PageRank algorithm?

Google's John Mueller now says that Googlebot renders the pages it crawls to determine what content is on them. They do this because so many sites are now heavily JavaScript centric and this allows ...
4
votes
3answers
1k views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...
4
votes
2answers
168 views

Google SEO - Migrating website from a sub-directory of another website to its own domain name

At the present moment I have a website hosted on example.com/myWebsite, where example.com hosts in its root directory a different website. I have the domain example.net, which redirects to example.com/...
4
votes
2answers
40 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
4
votes
1answer
358 views

Why is the date displayed in the Google search result is 3 days older than the actual publication date?

Yesterday I published an article on my website, if I go on Google it says that it has been published 3 days ago. Why? I've WordPress, on a Centos 6,5 VM. System time is wrong of 20 min of my server, ...
4
votes
1answer
237 views

Using rel=canonical and noindex in a 1-n partners environment

We sell complete sites including the domains to several partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original but the ...
4
votes
0answers
553 views

How Googlebot discovers new website? [duplicate]

I have purchased a domain and uploaded with HTML Page and When I search in Google it shows my website. Now my question is How Google will discover my new website? I have not used sitemap,robot....
4
votes
1answer
502 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
3
votes
1answer
73 views

Replace all images when search engine crawler visits site

I have written code that will detect when a user is a search engine and i dont include css files when this happens to save bandwidth. I was also thinking i could replace ALL images on the page with a ...
3
votes
5answers
506 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
3
votes
3answers
744 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my page,...
3
votes
2answers
48 views

Googlebot is still attempting to crawl old content

I have content that was deleted several years ago and from time to time Googlebot still attempts to access those pages, filling up my logs with lots of 404, making the 'real' problems harder to find ...
3
votes
1answer
230 views

Changing URLs in Google's Index

I recently submitted a sitemap.xml in Google's Webmaster Tools and some of the pages already got indexed. The problem now is, that I cleaned my URLs with MOD_REWRITE in the .htaccess file afterwards ...
3
votes
1answer
115 views

Will Google follow link if anchor tag content is blocked?

This is a followup on a question I was asking here: Cropping images & SEO Basically, if I have something like this: <a href="<Full_Res_Image>"><img src="<Cropped_Image>"&...
3
votes
1answer
728 views

Google is not indexing my site although it is crawling it

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message: URL and ...
3
votes
2answers
317 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
3
votes
1answer
561 views

Relative URLs confuse Google crawler

I'm having some odd problems that are apparently being generated by Google crawler. Bing isn't these errors, so it's kind of confusing. The problem seems to be because of relative links. Google ...
3
votes
2answers
118 views

Is offering Google Bot a lite version of site skin considered cloaking?

If you have multiple templates a user can choose from on a site (like a forum), and you offer Google Bot a "lite" skin, is this considered cloaking? Or is this considered okay? The actual content ...
3
votes
2answers
1k views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
3
votes
2answers
2k views

What does Google do with indexed pages returning 403?

I guess that Google removes already indexed pages that are now returning the HTTP error 403. Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
3
votes
1answer
44 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
3
votes
2answers
278 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
3
votes
1answer
112 views

Does every page HAVE to be mobile friendly even if its almost unimportant?

I received the following regarding a URL of a page recommending a feature that only applies to desktop users: Faulty Redirect URLs redirect to an irrelevant landing page instead of the ...
3
votes
3answers
116 views

Facets: Googlebot encountered extremely large numbers of links on your site

We are encountering the following issue: Googlebot encountered extremely large numbers of links on your site. The site is an ecommerce site and has around 12M pages indexed in Google. From the ...
3
votes
3answers
59 views

Indexed webpages stopped rising, while Submited keep rising normally

Last months I have noticed a strange behaviour in Google's Webmaster Tools. Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while ...
3
votes
1answer
339 views

How to tell Google my content will expire after a certain period? [duplicate]

Right now I'm facing issues with 404 (Not Found) errors, because my content will deleted automatically after two months. I cannot change this behavior, because it is a real estate web site and once it ...
3
votes
2answers
1k views

Foolproof way to ensure Google news pulls the correct image for its thumbnails?

Google news results have an acompanying thumbnail next to articles that show up in the results. If Google's crawler can't find a thumbnail to pull from our site, it uses its next best guess from ...
3
votes
2answers
1k views

Is there any benefit to put rel='nofollow' on mailto links?

Since bots have a fixed budget to spend to crawl websites on a regular basis, does it makes sense to mark mailto links with rel='nofollow' to make sure them don't spend time on these (especially if ...
3
votes
1answer
549 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady pace,...
3
votes
1answer
37 views

Can I “cloak” for crawlers and not use session id parameters and redirects to check for cookies?

I track my users using GET parameters to be able to do it even if they don't allow the use of cookies. And to do that, I need to do 2 redirections the first time the user hits my page. I am having ...
3
votes
1answer
32 views

Google blocks map API, GWT shows severity high, will this cause SEO problems?

Google blocked map API, GWT showed severity high. What is the effect on SEO?
3
votes
1answer
95 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
3
votes
1answer
205 views

Does Google index images declared using 'srcset'?

As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
3
votes
1answer
964 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
3
votes
1answer
225 views

Is having content on the 'above the fold' area of webpage good for SEO?

I have content (article) which is visible only when scrolled down. I have dedicated the 'above the fold' area of all my article webpages for visuals - graphics and H1 (header 1). Is there any ...
3
votes
1answer
449 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
3
votes
1answer
110 views

Odd/faulty page links being served to Googlebot

Yahoo recently bought Tumblr, where I write fairly long form scientific pieces. I noticed recently that a new script had been added to the end of pages, and it seems to be interacting badly with the ...
3
votes
1answer
85 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
3
votes
1answer
461 views

Can I use meta refresh instead of 301 redirect?

I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42. Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...
3
votes
1answer
195 views

Google 404 errors on webmaster page

I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
3
votes
5answers
587 views

SEO problems with multilanguage site [duplicate]

Possible Duplicate: How can I get search engines to crawl my site and see a localised view of my data? I have recently started a CMS project in PHP, the site will be multilingual. Assumptions ...
3
votes
4answers
748 views

Personalized content (authenticated with a cookie) and impact on SEO

We have a big website with a lot of content, that is doing pretty well in terms of SEO. Google likes the site, visitors like it as well. Our problem is that there is a lot of content that would only ...
3
votes
2answers
179 views

Google crawl rate is climbing but still no indexing?

This might belong on another stack site but idk. Soooo.... A year ago I started two sites that I didn't want indexed by Google or any other search engine since both sites were being developed based on ...
3
votes
1answer
177 views

Will Google index a React JS SPA if I can't add a rewrite rule?

The React SPA is set up to dynamically pull content from a server based on the URL. There is an index.html file sitting in a directory that routes to the correct content from the page requests. The ...