Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
30 votes
4 answers
13k views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
28 votes
1 answer
5k views

Why does my IP address become Google's IP address when using Chrome on the mobile?

I am tracking every member's geolocation (using db-ip) and everything is fine except mobile phones with the Chrome browser. They always give me the result US Mountain View California ip:66.249.xxx....
  • 475
25 votes
3 answers
12k views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
19 votes
3 answers
5k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
17 votes
7 answers
10k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented Googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
  • 341
13 votes
3 answers
6k views

Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?

I've seen code like for preventing Google from indexing part of a page: <p>This is a paragraph that will be indexed by Google.</p> <!--googleoff: index--> <p>This is a ...
13 votes
1 answer
4k views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
12 votes
2 answers
7k views

What is .well-known/traffic-advice directory?

I'm getting lots of requests from Googlebot on the .well-known/traffic-advice directory on my server. What is it and what are they looking for? Should I add something to this directory, and if yes ...
  • 253
12 votes
2 answers
1k views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
  • 2,080
12 votes
4 answers
10k views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
  • 191
12 votes
4 answers
943 views

Does Googlebot support TLS v1.2? If not, what will my TLS v1.2-only site look like to it?

I currently redirect all non-HTTPS connections to HTTPS, and support TLS v1, TLS v1.1, and TLS v1.2. I reconfigured my testing server to accept only a very limited and secure cipher suite and TLSv1.2 ...
  • 2,480
12 votes
2 answers
611 views

How is Googlebot finding URLs that are only visible to authenticated users?

Here is one of my customers, performing some action after having logged in to his account. The unique token is simply an encrypted user id + timestamp. 94.254.xxx.xxx - - [02/Jul/2011:22:25:46 +...
  • 271
12 votes
1 answer
11k views

link rel="alternate", multiple languages and canonical urls

Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain. On some ...
  • 233
10 votes
3 answers
2k views

Do Search Engine Spider's submit POST forms?

If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form? For example, if I were to have 4 ...
10 votes
1 answer
2k views

How does "Noindex:" in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
10 votes
1 answer
741 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
  • 201
10 votes
1 answer
7k views

New Google Search Console does not load any resources such as CSS and JS while it worked fine in the old Search Console tool [duplicate]

When using URL inspection and Live Test in the new Google Search Console the Googlebot does not load most resources (Javascript, CSS and Images). This results in the Search Console telling me my pages ...
  • 101
9 votes
2 answers
587 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9 votes
1 answer
316 views

Does Googlebot support SPDY?

When SPDY is available for a site, does Googlebot use it? If so, sites which are SPDY enabled would appear faster to Googlebot.
  • 549
9 votes
2 answers
6k views

Base64 encoded images and availability of their metadata for Googlebot

If I embed an image into a page as an img-src with base64 data URI, are image's metadata (EXIF, IPTC, XMP) still available for Google's imagebot?
  • 9,754
9 votes
1 answer
6k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
  • 245
9 votes
1 answer
1k views

Should I ping to Google sitemap-index.xml or only the sitemap file that has been modified?

I have a sitemap-index.xml with 7 files. After updating one of the 7 with new content should I ping Google with the index file or with the updated file? In my index I also use lastmod so Google can ...
8 votes
5 answers
1k views

Is it worthwhile to pay for boosting SEO for a new wesite?

I have come across a couple of websites today that claim to boost your website index in a matter of days. There are 1000s of reviews and all of them seem quite positive showing off results and stuff. ...
8 votes
3 answers
6k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
  • 1,412
8 votes
4 answers
5k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
  • 407
8 votes
1 answer
10k views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record (...
8 votes
2 answers
95 views

Informing Googlebot for deprecated pages

I publish timetables in my website. For example last year I published Number 2 bus Summer 2013 timetable. I has pretty good ranking on Google SERPs for number 2 bus timetable But this year I added a ...
  • 405
8 votes
1 answer
2k views

Structured Data Tool doesn't see JavaScript rendered content

Some structured data (namely price) of my page is rendered by JavaScript. According to fetching results in Webmaster tools Googlebot sees and renders it perfectly. However the structured data tool ...
user avatar
8 votes
1 answer
3k views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
8 votes
1 answer
429 views

Will implementing HSTS prevent Googlebot from seeing 301 permanent redirects from HTTP to HTTPS?

I did a migration from http:// to https:// for a big site with more than 2 million of URLs indexed on Google. As others have mentioned I'm also experiencing a bit of fluctuation on organic traffic (-...
8 votes
1 answer
1k views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
7 votes
5 answers
2k views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
7 votes
4 answers
3k views

Why is preventing Googlebot from crawling some pages on my site good for SEO?

Google says that it is effective to prevent Google Crawling Bot from crawling some useless pages in my site. https://developers.google.com/search/docs/advanced/guidelines/how-search-works https://...
7 votes
3 answers
2k views

How do I prevent Google from indexing text that is generated by JavaScript?

Google is executing the JavaScript on my password creator website and indexing the random passwords that get generated by the JavaScript. The random text that is generated could look spammy to Google ...
7 votes
2 answers
4k views

Googlebot crawls my page too often

I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot). But I have 1 article that have ...
7 votes
1 answer
2k views

Does htpasswd authentication prevent Google from crawling my site?

I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing. I currently ...
  • 268
7 votes
1 answer
1k views

How disqus comments (inside cross domain iframe) are indexed by google?

According to what I have read is that google doesn't index content from any cross domain iframe since it's the content of another website. So how does Disqus do it?
7 votes
3 answers
1k views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
7 votes
2 answers
6k views

Google-bot sees “Sorry, we have no imagery here” on pages with Google Maps

I have a site with Google Maps on most of the pages. When inspecting content keywords in Google Webmaster tools, content keywords identified by Google-bot for the site include "imagery", "sorry" and "...
  • 173
7 votes
1 answer
3k views

Cookie consent overlay - Does it affect SEO?

I have a german website with Adsense integrated. To meet Google's compliance requirements, I ask users to accept cookies for personal advertising. To achieve this, I created an overlay which stays in ...
7 votes
4 answers
239 views

Why Google put a nofollow tag on their homepage?

Why did Google put a nofollow tag on their homepage? Find an ATM near you is a nofollow link. They own the content on the same domain name. If it is a different domain name then it may make sense as ...
  • 5,497
7 votes
3 answers
2k views

For webpages using jQuery, will the Googlebot index them on "page load" or after the "page ready" state?

Will the Googlebot index a webpage on "page load" or after the "page ready" state. The reason I'm asking is because I'm trying to change my h1 and h2 tags using jQuery on page Load.
6 votes
3 answers
6k views

How do I stop Google from indexing AJAX calls within Javascript?

We are getting a number of odd results come up on Google EG: http://www.somedomain.com/somepage/AjaxFunction.aspx?stuff=XXX&other=XXX When I looked at "somepage" the ajax function is not ...
6 votes
1 answer
732 views

Robots.txt and pattern matching

Adding this to my robots.txt User-agent: * Disallow: /*action=*$ How does robots not recognizing wild cards handle this?
6 votes
3 answers
3k views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...
  • 531
6 votes
1 answer
658 views

301 and no-index tag don't work

Almost 3 months ago I've put no-index tag to about 3,000 pages on my website and 301 redirect (which is permanent redirect) for almost 15,000 pages. (The all site is about 50,000 pages) And yet as ...
  • 661
6 votes
1 answer
707 views

How to interpret Google policy on crawling and indexing search results pages?

I just learned that we can be penalized for letting Google crawl and index our database search results. https://webmasters.stackexchange.com/a/55599/33777 Question: If they don't want to list Yellow ...
  • 329
6 votes
3 answers
2k views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
  • 133
6 votes
2 answers
355 views

How to keep crawler robots from indexing my page when there is a query string present in the URL?

I have a /products page. In that page, users may use filters to select the products they want to see. Those filters will work using query strings like: /products?brands=X,Y&size=1,2 and so on. I ...
6 votes
2 answers
1k views

First time message/popup in site (by cookie), Googlebot and SEO

I have a site (based on dynamic content by JS) and I want to trigger a first time popup to users in order to make them register to the newsletter (it will be cookies based). Is there a way to do that ...
  • 63

1
2 3 4 5
19