Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
0answers
59 views

Does Google still crawl pages with dynamically changing content like news feeds?

Some websites like the following have a news feed on their homepage that provides the majority of content, which is an increasingly popular feature: https://www.producthunt.com/ https://www.tripoto....
2
votes
3answers
126 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
1
vote
2answers
50 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?
0
votes
2answers
104 views

SEO effects of H1 carousel slider

I maintain a website for a law firm. Originally their main desired keyword was "zoning attorneys." So I created a small banner that has an h1 "ZONING ATTORNEYS" on every page with the attorneys' names ...
2
votes
1answer
56 views

Can you use Fetch as Google to get a page crawled and indexed on a domain you don't control?

I love Google Webmaster Fetch as Google to get the page crawled for freshness. Is there a way to do something similar on a domain I don't control? I can't add the specific domain in Google Webmaster ...
2
votes
1answer
149 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
0
votes
1answer
21 views

How to Block Spam Referral Traffic in Google Analytic? [duplicate]

I have noticed so many spam referral traffic on my website on Google Analytic. I have tried to block them with the help of filters but it could not help. I know there is also a way to block it with ...
7
votes
5answers
595 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
3
votes
1answer
95 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
0
votes
1answer
20 views

What client on webpagetest.org is equivalent to googlebot client?

I made various speed tests to my website via webpagetest.org and I used a mobile network in California (which I believe is the closest one to google) to measure the speed of my site. It reports the ...
3
votes
2answers
163 views

Does Google index AJAX websites after running the JavaScript?

As I understand Google depreciated its ajax crawling scheme they came up with back in 2009. I run single page app powered by Anguar frontend with push state navigation and my question is around ...
0
votes
0answers
10 views

Spiders cannot crawl all my website pages [duplicate]

I'm struggling with googlebot webpages indexing; in particular I have a page with many links to other subpages that aren't indexed. The strange thing is that, if you submit in any online crawler the ...
1
vote
1answer
31 views

What Gzip compression level can safely be used for sitemap files?

I can use PHP's gzencode function to set the compression level of data to compress and I want to compress my sitemap files then submit them to google. Currently my compression level is set to 2 ...
1
vote
0answers
33 views

Does fetch and render + submit to index interrupt Google's crawl period?

If you resubmit your site to the index in Webmaster Tools, will pending crawls get cancelled? I ask because a client's site has been up for 72 hours now and has not been crawled. I submitted to index ...
2
votes
1answer
81 views

How to visually see frequency of GoogleBot / BingBot access of my website from apache access logs (per 15 mins / etc)?

I am experiencing high traffic on my website at certain times of the data, at regular intervals. I am trying to determine whether GoogleBot or BingBot access of my website correlates with the high ...
3
votes
1answer
56 views

Fetch as Google says my page is “Temporary Unreachable”

When looking into it in WebmasterTools, I saw that there is a "Fetch as Google - Temporarily Unreachable Issue". It is for an image, and font. How do I resolve this issue?
1
vote
3answers
56 views

Does Googlebot use hreflang for link discovery?

Does anyone know if Googlebot will follow and index web pages that are only listed in the head of a HTML page with a link-hreflang element? <link hreflang="fr" rel="alternate" href="http://example....
0
votes
0answers
47 views

Why is hidden content not always indexed?

I've read several blogs discussing the 'hidden content' issue. This article experimented with several options. It concludes that Google DOES index hidden content. The issues is also discussed here. ...
1
vote
2answers
60 views

Google Structured Data and sitemap resubmit?

I've submitted sitemap in google search console a few weeks ago and slowly being indexed. I just implemented Google Structure Data to all the pages. How strong does it for the data to show up in ...
4
votes
2answers
479 views

Is it bad for SEO to submit a partially finished site to Google?

Hypothetical Situation 1: I have a plan for an amazing large-scale website with lot's of really cool features, animations and interactivity. This will take a several months to completely develop. ...
4
votes
2answers
598 views

Hide joyride (intro javascript) for Googlebot

I am using a joyride script for new visitors. Example here: http://foundation.zurb.com/sites/docs/v/5.5.3/components/joyride.html After the vistior is done with the intro, a cookie will be created ...
4
votes
2answers
40 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
1
vote
2answers
60 views

Why can't I find a certain text on my site in Google?

I can't find certain text on my site in Google, even if I'm searching specifically on the page that this text in on: site:mysite.com/page.html hidden text I tried searching just for "site:mysite....
1
vote
2answers
67 views

How Google crawler behaves when the HTML page is 1.6MB? [duplicate]

Is it possible that Google crawler doesn't fetch all the content on the page when it's size is 1.6 MB? How does Google treat big documents?
2
votes
2answers
44 views

Will googlebot misinterpret my site if I had no spacing between anchor tags?

Some sections of my website involve displaying a list of photo thumbnails. The source code to that is similar to the following when condensed: <div> <a href="1">1</a> <a href="2"&...
1
vote
1answer
59 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
1
vote
1answer
124 views

How to update the date on Google search results?

For a blog entry, I have this meta tags: <meta property="article:published_time" content="2015-09-07T21:54:43+00:00" /> <meta property="article:modified_time" content="2015-11-13T13:28:21+00:...
2
votes
2answers
130 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
1
vote
2answers
83 views

Why can't Google Bot access my site?

I repeatedly get an warning/error message from Google as follows: Googlebot can't access your site I have checked, edited, removed, replaced the robots.txt file on the site all to no avail. ...
0
votes
0answers
273 views

SEO on ui-router states of AngularJS SPA

I have a simple SPA created with AngularJS, you can see it running here: www.consami.com.br. I was using Prerender.io to load a pure HTML version for crawlers, however some pages were not indexed by ...
1
vote
2answers
47 views

Crawl Budget economy: what to do with URLs with tracking parameters?

So, a site has inbound links to a lot of important pages and these links have URL parameters. These URL parameters are basically used just for tracking purposes, similar to the way Amazon links to its ...
0
votes
0answers
38 views

access.log requests from UA “Google favicon”…? What do these mean?

I have a bunch of these in my nginx log file, on a range of URLs: on 66.102.9.23 - - [12/Nov/2015:10:40:25 +0000] "GET /myadminfolder/page.html HTTP/1.1" 200 1247 1800 "-" "Google favicon" "...
0
votes
0answers
24 views

How should I render a widget with many links inside a blog so google don't penalize the blog?

We run a service (will not mention the URL, as this is not a pitch) and need advice on Google-viewing-links matters. The service is for bloggers and allows them to create a "linkup" inside their blog ...
0
votes
0answers
13 views

Google not crawling all pages [duplicate]

I have over 200,000 pages but only about 23,000 pages are crawled by Google as is evident when i search for site:mywebsite.com and the result comes as About 22,900 results (0.18 seconds) I have over ...
3
votes
2answers
279 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
0
votes
1answer
44 views

Why does Googlebot-Mobile crawl desktop content?

Identical desktop and mobile URLs, no subdomain or sub-folder: Desktop URL site-name.com <link rel="canonical" href="index.php?/..."> Mobile URL site-name.com <link rel="canonical" href="...
1
vote
1answer
135 views

Does Google bot crawl dynamically created iframes? [duplicate]

The situation looks as followed: User includes Javascript snippet into his page. Javascript creates iFrame dynamically and appends it to page. iFrame has static content. Is Google bot smart enough ...
1
vote
1answer
29 views

Serving a non-responsive css to ie8 and lower

We are rewritting our website to a responsive design (and maybe RESS in the future). Analytics figures impose us to support some old browsers (ie7 and ie8). We have 2 CSS: site.css: full css with ...
1
vote
0answers
45 views

Does 'thin-content' content on some pages affect overall site SEO?

I have an UGC Q&A site. Due to it's nature is inevitable to see some 'thin' or low quality content in some posts. Will there be any sitewide impact in SEO for the other high quality pages? If yes ...
9
votes
2answers
326 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
1
vote
1answer
61 views

Session ID Creating 404 Errors

Problem: Website contains URLs with parameters Session ID gets inserted into URLs when Googlebot crawls page with parameter (sorting) options Session ID is created and then expires, creating a 404 ...
0
votes
1answer
52 views

IP Geolocation and Googlebot

For a site that displays events near a city based on the user's IP address, how should Googlebot (or other crawlers) be handled? The site has many different pages and most of them use location-based ...
2
votes
3answers
101 views

A/B Testing A Site Redesign Without Google Penalties

I am redesigning our job board website to be responsive. Rather than just replace the existing site with the responsive site, I want to A/B test the new responsive site alongside the existing site so ...
2
votes
2answers
150 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
2
votes
1answer
69 views

English content on International page templates: how to mark it up with lang=“” attributes?

So, I have translated page templates to various languages so it's easier for users to pick their language for easier navigating and understanding the site. The pages have similar markup to this ...
1
vote
2answers
76 views

Bandwidth saturation due to Googlebot search for spam pages

I recently had to update the hosting plan of a WordPress site I run due to bandwidth excess, which was already quite high. I've installed a security plugin that keeps log of 404 errors. From here I ...
0
votes
0answers
8 views

Incorrect Titles, Descriptions and Redirects in Google [duplicate]

I'm at a loss for why Google is indexing my site incorrectly. I just uploaded a new version of the site last night and "fetched as google" last night as well. Also uploaded sitemap. www.lemonsevens....
1
vote
1answer
32 views

Is it harmful to block google while I continue to build content?

I have been blocking all IPs to my website (except my own) while it is in development. But now I am at a point, where I want other people to be able to see it, but I still don't want googlebot on it ...
2
votes
1answer
40 views

deindexing a whole domain with robots.txt created problems in google webmaster tools

In my robots.txt for my domain that just hosts images, I used the following lines: user-agent: * noindex: / I thought I was doing search engines a wonderful favor by not having them crawl through ...
0
votes
2answers
163 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...