Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

3
votes
5answers
494 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
3
votes
3answers
402 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my ...
3
votes
1answer
87 views

Will Google follow link if anchor tag content is blocked?

This is a followup on a question I was asking here: Cropping images & SEO Basically, if I have something like this: <a href="<Full_Res_Image>"><img ...
3
votes
2answers
227 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
3
votes
1answer
386 views

Relative URLs confuse Google crawler

I'm having some odd problems that are apparently being generated by Google crawler. Bing isn't these errors, so it's kind of confusing. The problem seems to be because of relative links. Google ...
3
votes
1answer
500 views

Google is not indexing my site although it is crawling it

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message: URL and ...
3
votes
2answers
114 views

Is offering Google Bot a lite version of site skin considered cloaking?

If you have multiple templates a user can choose from on a site (like a forum), and you offer Google Bot a "lite" skin, is this considered cloaking? Or is this considered okay? The actual content ...
3
votes
2answers
1k views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
3
votes
2answers
1k views

What does Google do with indexed pages returning 403?

I guess that Google removes already indexed pages that are now returning the HTTP error 403. Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
3
votes
1answer
95 views

Does every page HAVE to be mobile friendly even if its almost unimportant?

I received the following regarding a URL of a page recommending a feature that only applies to desktop users: Faulty Redirect URLs redirect to an irrelevant landing page instead of the ...
3
votes
3answers
83 views

Facets: Googlebot encountered extremely large numbers of links on your site

We are encountering the following issue: Googlebot encountered extremely large numbers of links on your site. The site is an ecommerce site and has around 12M pages indexed in Google. From the ...
3
votes
3answers
54 views

Indexed webpages stopped rising, while Submited keep rising normally

Last months I have noticed a strange behaviour in Google's Webmaster Tools. Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while ...
3
votes
1answer
252 views

How to tell Google my content will expire after a certain period? [duplicate]

Right now I'm facing issues with 404 (Not Found) errors, because my content will deleted automatically after two months. I cannot change this behavior, because it is a real estate web site and once it ...
3
votes
2answers
844 views

Foolproof way to ensure Google news pulls the correct image for its thumbnails?

Google news results have an acompanying thumbnail next to articles that show up in the results. If Google's crawler can't find a thumbnail to pull from our site, it uses its next best guess from ...
3
votes
1answer
144 views

Is having content on the 'above the fold' area of webpage good for SEO?

I have content (article) which is visible only when scrolled down. I have dedicated the 'above the fold' area of all my article webpages for visuals - graphics and H1 (header 1). Is there any ...
3
votes
1answer
236 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
3
votes
1answer
81 views

Odd/faulty page links being served to Googlebot

Yahoo recently bought Tumblr, where I write fairly long form scientific pieces. I noticed recently that a new script had been added to the end of pages, and it seems to be interacting badly with the ...
3
votes
1answer
80 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
3
votes
1answer
406 views

Can I use meta refresh instead of 301 redirect?

I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42. Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...
3
votes
1answer
189 views

Google 404 errors on webmaster page

I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
3
votes
5answers
505 views

SEO problems with multilanguage site [duplicate]

Possible Duplicate: How can I get search engines to crawl my site and see a localised view of my data? I have recently started a CMS project in PHP, the site will be multilingual. ...
3
votes
4answers
615 views

Personalized content (authenticated with a cookie) and impact on SEO

We have a big website with a lot of content, that is doing pretty well in terms of SEO. Google likes the site, visitors like it as well. Our problem is that there is a lot of content that would only ...
3
votes
2answers
159 views

Google crawl rate is climbing but still no indexing?

This might belong on another stack site but idk. Soooo.... A year ago I started two sites that I didn't want indexed by Google or any other search engine since both sites were being developed based on ...
3
votes
1answer
170 views

Webmaster Tools shows URL errors linked from http://com.example.www/:http

I have a bog standard WordPress 4.2.1 site. I recently added it to Webmaster Tools and it is showing me URL errors (404s) that are linked from a weird URL: http://com.example.www/:http (I have ...
3
votes
1answer
66 views

Is Google less likely to index URLs that don't match the directory structure of the page they are linked from?

Worded the question title the best I could. I have a website. The site has a category of articles - award winners. the URL to this category, a page with a list of winners, is: ...
3
votes
1answer
67 views

Is there a way to stop a keyword from appearing in Google Webmaster Tools?

I added a web site a while ago to Google's index and it contains lots of links with the text "Permalink" that, well, link to permalinks of different content/pages. Now "Permalink" is the #1 keyword ...
3
votes
2answers
182 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
3
votes
1answer
469 views

Google crawlers screen size

Imagine a I have two types of images (A and AAA(heavier)) for the same website: depending on the size of the screen it will show one image or other. Now I'm wondering what images (A or AAA) loads ...
3
votes
2answers
98 views

Directing crawlers to content in language per language sub-domain

I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
3
votes
1answer
507 views

Webmaster Tools show the “Total Indexed” number to be zero

I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
3
votes
1answer
98 views

Why is the date displayed in the Google search result is 3 days older than the actual publication date?

Yesterday I published an article on my website, if I go on Google it says that it has been published 3 days ago. Why? I've WordPress, on a Centos 6,5 VM. System time is wrong of 20 min of my server, ...
2
votes
4answers
836 views

Google bots are severely affecting site performance

I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on ...
2
votes
2answers
459 views

Is hiding content with JavaScript or “text-indent: -9999px” bad for SEO?

So apparently hiding content using "display: none" is bad for SEO and seen by googlebot as being deceptive. This according to a lot of the posts I read online and questions even on this site. But ...
2
votes
3answers
369 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
2
votes
1answer
514 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
2
votes
2answers
84 views

Will multilingual content on same page works to avoid content duplication

We have a website, which is having old/new vehicle deals for purchase. We are getting these data from third party website (legally) and all the pages are in English language. So, Here our issue is ...
2
votes
4answers
1k views

Does updating site too frequently hurt SEO rankings?

I recently updated my site, like 3-4 times in 2 months. At first everything was good, (i.e., Google was updating my pages' SERP content in 2-3 days). But now I did kind of a major change, I switched ...
2
votes
3answers
2k views

How much time does Googlebot take to crawl the page after submitting it to index

I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority. I've a page linking to newly added jobs ...
2
votes
3answers
185 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
2
votes
1answer
57 views

Is google penalizing SEO of websites with non device-width viewports?

These articles make it look like they are: https://support.google.com/webmasters/answer/6101188?hl=en https://developers.google.com/web/fundamentals/layouts/rwd-fundamentals/set-the-viewport This ...
2
votes
2answers
76 views

Using table tag to structure a web page decrease referencing quality? [duplicate]

I am student in web programing. I use massively <table> ... </table> to structure my web page. My professor once told me it could decrease the referencing quality for my websites, so I ...
2
votes
2answers
130 views

What's the smallest page size for Googlebot to not go “Soft 404”?

Google appears to have two documents for Soft 404: https://support.google.com/webmasters/answer/181708?hl=en&ctx=tltp https://support.google.com/webmasters/answer/2409443?ctx=MCE&ctx=S4 ...
2
votes
1answer
54 views

Does Googlebot determine the value of a link by its position on the page?

It's well-known that the SEO value of a website basically depends on how much it has been referenced, linked, and shared. From Googlebot's point of view, is a link at the top of a page more valuable ...
2
votes
2answers
178 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
2
votes
3answers
882 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
2
votes
1answer
300 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
2
votes
2answers
1k views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
2
votes
1answer
46 views

How to present videos for Google Bot

i have a website which streams Flash Videos. For Google i have a special page due to using AJAX for normal Users (see articles on "_escaped_fragment_" for this). What i want to know is how i should ...
2
votes
3answers
161 views

Backlinks: How does a company's name as the anchor text help with SEO?

Often I see footer links like: SEO by COMPANY NAME How does this help with SEO other than helping a company rank for "Company Name"? When nearby words like "SEO by" get picked up by a search engine ...