Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

0
votes
1answer
9 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
3
votes
1answer
28 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. ...
0
votes
1answer
8 views

What client on webpagetest.org is equivalent to googlebot client?

I made various speed tests to my website via webpagetest.org and I used a mobile network in California (which I believe is the closest one to google) to measure the speed of my site. It reports the ...
0
votes
1answer
48 views

How google index the Ajax websites?

As I understand google depreciated its ajax crawling scheme they came up with back in 2009. I run single page app powered by Anguar frontend with push state navigation and my question is around ...
0
votes
0answers
9 views

Spiders cannot crawl all my website pages [duplicate]

I'm struggling with googlebot webpages indexing; in particular I have a page with many links to other subpages that aren't indexed. The strange thing is that, if you submit in any online crawler the ...
1
vote
1answer
13 views

What Gzip compression level can safely be used for sitemap files?

I can use PHP's gzencode function to set the compression level of data to compress and I want to compress my sitemap files then submit them to google. Currently my compression level is set to 2 ...
1
vote
0answers
24 views

Does fetch and render + submit to index interrupt Google's crawl period?

If you resubmit your site to the index in Webmaster Tools, will pending crawls get cancelled? I ask because a client's site has been up for 72 hours now and has not been crawled. I submitted to index ...
3
votes
1answer
52 views

How to visually see frequency of GoogleBot / BingBot access of my website from apache access logs (per 15 mins / etc)?

I am experiencing high traffic on my website at certain times of the data, at regular intervals. I am trying to determine whether GoogleBot or BingBot access of my website correlates with the high ...
3
votes
1answer
27 views

Fetch as Google says my page is “Temporary Unreachable”

When looking into it in WebmasterTools, I saw that there is a "Fetch as Google - Temporarily Unreachable Issue". It is for an image, and font. How do I resolve this issue?
1
vote
3answers
49 views

Does Googlebot use hreflang for link discovery?

Does anyone know if Googlebot will follow and index web pages that are only listed in the head of a HTML page with a link-hreflang element? <link hreflang="fr" rel="alternate" ...
0
votes
0answers
42 views

Why is hidden content not always indexed?

I've read several blogs discussing the 'hidden content' issue. This article experimented with several options. It concludes that Google DOES index hidden content. The issues is also discussed here. ...
2
votes
1answer
31 views

Google Structured Data and sitemap resubmit?

I've submitted sitemap in google search console a few weeks ago and slowly being indexed. I just implemented Google Structure Data to all the pages. How strong does it for the data to show up in ...
4
votes
2answers
434 views

Is it bad for SEO to submit a partially finished site to Google?

Hypothetical Situation 1: I have a plan for an amazing large-scale website with lot's of really cool features, animations and interactivity. This will take a several months to completely develop. ...
4
votes
2answers
572 views

Hide joyride (intro javascript) for Googlebot

I am using a joyride script for new visitors. Example here: http://foundation.zurb.com/sites/docs/v/5.5.3/components/joyride.html After the vistior is done with the intro, a cookie will be created ...
4
votes
2answers
29 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
1
vote
2answers
54 views

Why can't I find a certain text on my site in Google?

I can't find certain text on my site in Google, even if I'm searching specifically on the page that this text in on: site:mysite.com/page.html hidden text I tried searching just for ...
1
vote
2answers
44 views

How Google crawler behaves when the HTML page is 1.6MB? [duplicate]

Is it possible that Google crawler doesn't fetch all the content on the page when it's size is 1.6 MB? How does Google treat big documents?
2
votes
2answers
41 views

Will googlebot misinterpret my site if I had no spacing between anchor tags?

Some sections of my website involve displaying a list of photo thumbnails. The source code to that is similar to the following when condensed: <div> <a href="1">1</a> <a ...
1
vote
1answer
37 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
1
vote
1answer
63 views

How to update the date on Google search results?

For a blog entry, I have this meta tags: <meta property="article:published_time" content="2015-09-07T21:54:43+00:00" /> <meta property="article:modified_time" ...
2
votes
2answers
123 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
1
vote
2answers
75 views

Why can't Google Bot access my site?

I repeatedly get an warning/error message from Google as follows: Googlebot can't access your site I have checked, edited, removed, replaced the robots.txt file on the site all to no avail. ...
0
votes
0answers
92 views

SEO on ui-router states of AngularJS SPA

I have a simple SPA created with AngularJS, you can see it running here: www.consami.com.br. I was using Prerender.io to load a pure HTML version for crawlers, however some pages were not indexed by ...
0
votes
0answers
25 views

Crawl Budget economy: what to do with URLs with parameters?

So, a site has inbound links to a lot of important pages and these links have URL parameters. These URL parameters are basically used just for tracking purposes, similar to the way Amazon links to its ...
0
votes
0answers
23 views

access.log requests from UA “Google favicon”…? What do these mean?

I have a bunch of these in my nginx log file, on a range of URLs: on 66.102.9.23 - - [12/Nov/2015:10:40:25 +0000] "GET /myadminfolder/page.html HTTP/1.1" 200 1247 1800 "-" "Google favicon" ...
0
votes
0answers
23 views

How should I render a widget with many links inside a blog so google don't penalize the blog?

We run a service (will not mention the URL, as this is not a pitch) and need advice on Google-viewing-links matters. The service is for bloggers and allows them to create a "linkup" inside their blog ...
0
votes
0answers
12 views

Google not crawling all pages [duplicate]

I have over 200,000 pages but only about 23,000 pages are crawled by Google as is evident when i search for site:mywebsite.com and the result comes as About 22,900 results (0.18 seconds) I have over ...
2
votes
2answers
108 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
0
votes
1answer
29 views

Why does Googlebot-Mobile crawl desktop content?

Identical desktop and mobile URLs, no subdomain or sub-folder: Desktop URL site-name.com <link rel="canonical" href="index.php?/..."> Mobile URL site-name.com <link rel="canonical" ...
0
votes
1answer
58 views

Does Google bot crawl dynamically created iframes? [duplicate]

The situation looks as followed: User includes Javascript snippet into his page. Javascript creates iFrame dynamically and appends it to page. iFrame has static content. Is Google bot smart enough ...
1
vote
1answer
25 views

Serving a non-responsive css to ie8 and lower

We are rewritting our website to a responsive design (and maybe RESS in the future). Analytics figures impose us to support some old browsers (ie7 and ie8). We have 2 CSS: site.css: full css with ...
1
vote
0answers
38 views

Does 'thin-content' content on some pages affect overall site SEO?

I have an UGC Q&A site. Due to it's nature is inevitable to see some 'thin' or low quality content in some posts. Will there be any sitewide impact in SEO for the other high quality pages? If yes ...
8
votes
2answers
254 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
1
vote
1answer
47 views

Session ID Creating 404 Errors

Problem: Website contains URLs with parameters Session ID gets inserted into URLs when Googlebot crawls page with parameter (sorting) options Session ID is created and then expires, creating a 404 ...
0
votes
1answer
42 views

IP Geolocation and Googlebot

For a site that displays events near a city based on the user's IP address, how should Googlebot (or other crawlers) be handled? The site has many different pages and most of them use location-based ...
1
vote
3answers
69 views

A/B Testing A Site Redesign Without Google Penalties

I am redesigning our job board website to be responsive. Rather than just replace the existing site with the responsive site, I want to A/B test the new responsive site alongside the existing site so ...
1
vote
2answers
101 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: ...
2
votes
1answer
64 views

English content on International page templates: how to mark it up with lang=“” attributes?

So, I have translated page templates to various languages so it's easier for users to pick their language for easier navigating and understanding the site. The pages have similar markup to this ...
1
vote
2answers
51 views

Bandwidth saturation due to Googlebot search for spam pages

I recently had to update the hosting plan of a WordPress site I run due to bandwidth excess, which was already quite high. I've installed a security plugin that keeps log of 404 errors. From here I ...
0
votes
0answers
8 views

Incorrect Titles, Descriptions and Redirects in Google [duplicate]

I'm at a loss for why Google is indexing my site incorrectly. I just uploaded a new version of the site last night and "fetched as google" last night as well. Also uploaded sitemap. ...
1
vote
1answer
30 views

Is it harmful to block google while I continue to build content?

I have been blocking all IPs to my website (except my own) while it is in development. But now I am at a point, where I want other people to be able to see it, but I still don't want googlebot on it ...
2
votes
1answer
32 views

deindexing a whole domain with robots.txt created problems in google webmaster tools

In my robots.txt for my domain that just hosts images, I used the following lines: user-agent: * noindex: / I thought I was doing search engines a wonderful favor by not having them crawl through ...
0
votes
2answers
118 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
3
votes
1answer
52 views

Should I remove Google indexed pages before requesting a review after thin content manual action

Following on from my previous question - NOINDEX large number of pages to fix Google thin content manual action? Firstly, If I noindex all pages causing the manual action, is that enough for me to ...
5
votes
1answer
97 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
1
vote
0answers
41 views

Search engine ranking of paginated categories, subcategories and tag pages

Suppose we have an E-commerce domain with the following structure: domain/items - contains all items. domain/items/category - contains a narrower spectrum of results than above. ...
1
vote
1answer
53 views

SEO multiple ecommerce website running off the same codebase

I was wondering the about the effect a duplicate/similar codebase of products would have on a sites SEO across 190 domains. We have a dealer group with 190 for a certain class of products e.g. ...
1
vote
2answers
93 views

Issuu - PDF - Duplicated Content

I often write newsletters and publish them on my own site as PDF's. To potentially drive more traffic to my site, I was thinking about publishing the same newsletter files through sites like Scribd ...
0
votes
0answers
41 views

Odd requests from AdsBot-Google which re-include the domain name

It appears that since we started an AdWords campaign, our servers get tons of bogus requests from AdsBot-Google that for some reason include our own domain name in them. Example, if our website has ...
0
votes
2answers
83 views

Setting a preferred domain in Google webmaster tools

I have a site like "example.com". Questions 1: I can access to my site through "example.com" and www.example.com in web browser. Both non www and www version redirects to example.com. So, is it ...