Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
1answer
82 views

Google Master Tool: How allow external links in robots.txt

I have a problem with google rendering. Googlebot couldn't get all resources for this page. Because it's blocked by its own google robots.txt file. Here's an example of an external link. https://www....
3
votes
2answers
74 views

Increase in 5XX errors and 404 errors on GWT, are we under attack?

A lot of URLs are suddenly giving us 404s and 500s. On closer inspection, these URLs do not exist on our website and are being linked from random sources. As more URLs were inspected on GWT, a ...
3
votes
1answer
35 views

Google blocks map API, GWT shows severity high, will this cause SEO problems?

Google blocked map API, GWT showed severity high. What is the effect on SEO?
1
vote
1answer
28 views

What are the possible reasons for a sudden decrease in the number of images being indexed in my sitemap?

A number of sitemaps contain URLs, and a number of those URLs contain image URLs with captions, titles, etc. Suddenly over the past five days, there has been a drop in the number of images indexed by ...
0
votes
1answer
24 views

What affect, if any, would buying domain names with high volume search terms have on SEO?

Assume the following: I have a website: www.example.com. I want to rank www.example.com/searc-term-1 on the first page of Google. Would it help the SEO of www.example.com/searc-term-1 if I bought ...
1
vote
2answers
53 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
2answers
35 views

Are there any meta tags I can add to my website to make it more discoverable by search engines?

I am quite a beginner at website design, and in fact so much of an amateur that I don't like paying for hosting/higher search engine results. Are there any meta's that will simply make my website a ...
1
vote
0answers
61 views

Does Google still crawl pages with dynamically changing content like news feeds?

Some websites like the following have a news feed on their homepage that provides the majority of content, which is an increasingly popular feature: https://www.producthunt.com/ https://www.tripoto....
2
votes
3answers
134 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
1
vote
2answers
55 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?
0
votes
2answers
139 views

SEO effects of H1 carousel slider

I maintain a website for a law firm. Originally their main desired keyword was "zoning attorneys." So I created a small banner that has an h1 "ZONING ATTORNEYS" on every page with the attorneys' names ...
2
votes
1answer
73 views

Can you use Fetch as Google to get a page crawled and indexed on a domain you don't control?

I love Google Webmaster Fetch as Google to get the page crawled for freshness. Is there a way to do something similar on a domain I don't control? I can't add the specific domain in Google Webmaster ...
2
votes
1answer
167 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
0
votes
1answer
23 views

How to Block Spam Referral Traffic in Google Analytic? [duplicate]

I have noticed so many spam referral traffic on my website on Google Analytic. I have tried to block them with the help of filters but it could not help. I know there is also a way to block it with ...
7
votes
5answers
606 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
3
votes
1answer
111 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
0
votes
1answer
21 views

What client on webpagetest.org is equivalent to googlebot client?

I made various speed tests to my website via webpagetest.org and I used a mobile network in California (which I believe is the closest one to google) to measure the speed of my site. It reports the ...
3
votes
2answers
186 views

Does Google index AJAX websites after running the JavaScript?

As I understand Google depreciated its ajax crawling scheme they came up with back in 2009. I run single page app powered by Anguar frontend with push state navigation and my question is around ...
0
votes
0answers
10 views

Spiders cannot crawl all my website pages [duplicate]

I'm struggling with googlebot webpages indexing; in particular I have a page with many links to other subpages that aren't indexed. The strange thing is that, if you submit in any online crawler the ...
1
vote
1answer
35 views

What Gzip compression level can safely be used for sitemap files?

I can use PHP's gzencode function to set the compression level of data to compress and I want to compress my sitemap files then submit them to google. Currently my compression level is set to 2 ...
1
vote
1answer
37 views

Does fetch and render + submit to index interrupt Google's crawl period?

If you resubmit your site to the index in Webmaster Tools, will pending crawls get cancelled? I ask because a client's site has been up for 72 hours now and has not been crawled. I submitted to index ...
2
votes
1answer
83 views

How to visually see frequency of GoogleBot / BingBot access of my website from apache access logs (per 15 mins / etc)?

I am experiencing high traffic on my website at certain times of the data, at regular intervals. I am trying to determine whether GoogleBot or BingBot access of my website correlates with the high ...
3
votes
1answer
72 views

Fetch as Google says my page is “Temporary Unreachable”

When looking into it in WebmasterTools, I saw that there is a "Fetch as Google - Temporarily Unreachable Issue". It is for an image, and font. How do I resolve this issue?
1
vote
3answers
57 views

Does Googlebot use hreflang for link discovery?

Does anyone know if Googlebot will follow and index web pages that are only listed in the head of a HTML page with a link-hreflang element? <link hreflang="fr" rel="alternate" href="http://example....
1
vote
1answer
53 views

Why is hidden content not always indexed?

I've read several blogs discussing the 'hidden content' issue. This article experimented with several options. It concludes that Google DOES index hidden content. The issues is also discussed here. ...
1
vote
2answers
61 views

Google Structured Data and sitemap resubmit?

I've submitted sitemap in google search console a few weeks ago and slowly being indexed. I just implemented Google Structure Data to all the pages. How strong does it for the data to show up in ...
4
votes
2answers
486 views

Is it bad for SEO to submit a partially finished site to Google?

Hypothetical Situation 1: I have a plan for an amazing large-scale website with lot's of really cool features, animations and interactivity. This will take a several months to completely develop. ...
4
votes
2answers
602 views

Hide joyride (intro javascript) for Googlebot

I am using a joyride script for new visitors. Example here: http://foundation.zurb.com/sites/docs/v/5.5.3/components/joyride.html After the vistior is done with the intro, a cookie will be created ...
4
votes
2answers
42 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
1
vote
2answers
62 views

Why can't I find a certain text on my site in Google?

I can't find certain text on my site in Google, even if I'm searching specifically on the page that this text in on: site:mysite.com/page.html hidden text I tried searching just for "site:mysite....
1
vote
2answers
74 views

How Google crawler behaves when the HTML page is 1.6MB? [duplicate]

Is it possible that Google crawler doesn't fetch all the content on the page when it's size is 1.6 MB? How does Google treat big documents?
2
votes
2answers
44 views

Will googlebot misinterpret my site if I had no spacing between anchor tags?

Some sections of my website involve displaying a list of photo thumbnails. The source code to that is similar to the following when condensed: <div> <a href="1">1</a> <a href="2"&...
1
vote
1answer
64 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
1
vote
1answer
134 views

How to update the date on Google search results?

For a blog entry, I have this meta tags: <meta property="article:published_time" content="2015-09-07T21:54:43+00:00" /> <meta property="article:modified_time" content="2015-11-13T13:28:21+00:...
2
votes
2answers
131 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
1
vote
2answers
89 views

Why can't Google Bot access my site?

I repeatedly get an warning/error message from Google as follows: Googlebot can't access your site I have checked, edited, removed, replaced the robots.txt file on the site all to no avail. ...
0
votes
0answers
304 views

SEO on ui-router states of AngularJS SPA

I have a simple SPA created with AngularJS, you can see it running here: www.consami.com.br. I was using Prerender.io to load a pure HTML version for crawlers, however some pages were not indexed by ...
1
vote
2answers
51 views

Crawl Budget economy: what to do with URLs with tracking parameters?

So, a site has inbound links to a lot of important pages and these links have URL parameters. These URL parameters are basically used just for tracking purposes, similar to the way Amazon links to its ...
0
votes
0answers
44 views

access.log requests from UA “Google favicon”…? What do these mean?

I have a bunch of these in my nginx log file, on a range of URLs: on 66.102.9.23 - - [12/Nov/2015:10:40:25 +0000] "GET /myadminfolder/page.html HTTP/1.1" 200 1247 1800 "-" "Google favicon" "...
0
votes
0answers
24 views

How should I render a widget with many links inside a blog so google don't penalize the blog?

We run a service (will not mention the URL, as this is not a pitch) and need advice on Google-viewing-links matters. The service is for bloggers and allows them to create a "linkup" inside their blog ...
0
votes
0answers
13 views

Google not crawling all pages [duplicate]

I have over 200,000 pages but only about 23,000 pages are crawled by Google as is evident when i search for site:mywebsite.com and the result comes as About 22,900 results (0.18 seconds) I have over ...
3
votes
2answers
317 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
0
votes
1answer
45 views

Why does Googlebot-Mobile crawl desktop content?

Identical desktop and mobile URLs, no subdomain or sub-folder: Desktop URL site-name.com <link rel="canonical" href="index.php?/..."> Mobile URL site-name.com <link rel="canonical" href="...
1
vote
1answer
162 views

Does Google bot crawl dynamically created iframes? [duplicate]

The situation looks as followed: User includes Javascript snippet into his page. Javascript creates iFrame dynamically and appends it to page. iFrame has static content. Is Google bot smart enough ...
1
vote
1answer
30 views

Serving a non-responsive css to ie8 and lower

We are rewritting our website to a responsive design (and maybe RESS in the future). Analytics figures impose us to support some old browsers (ie7 and ie8). We have 2 CSS: site.css: full css with ...
1
vote
0answers
45 views

Does 'thin-content' content on some pages affect overall site SEO?

I have an UGC Q&A site. Due to it's nature is inevitable to see some 'thin' or low quality content in some posts. Will there be any sitewide impact in SEO for the other high quality pages? If yes ...
9
votes
2answers
331 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
1
vote
1answer
63 views

Session ID Creating 404 Errors

Problem: Website contains URLs with parameters Session ID gets inserted into URLs when Googlebot crawls page with parameter (sorting) options Session ID is created and then expires, creating a 404 ...
0
votes
1answer
53 views

IP Geolocation and Googlebot

For a site that displays events near a city based on the user's IP address, how should Googlebot (or other crawlers) be handled? The site has many different pages and most of them use location-based ...
2
votes
3answers
104 views

A/B Testing A Site Redesign Without Google Penalties

I am redesigning our job board website to be responsive. Rather than just replace the existing site with the responsive site, I want to A/B test the new responsive site alongside the existing site so ...