Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

3
votes
1answer
463 views

Can I use meta refresh instead of 301 redirect?

I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42. Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...
3
votes
1answer
195 views

Google 404 errors on webmaster page

I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
3
votes
5answers
592 views

SEO problems with multilanguage site [duplicate]

Possible Duplicate: How can I get search engines to crawl my site and see a localised view of my data? I have recently started a CMS project in PHP, the site will be multilingual. Assumptions ...
3
votes
4answers
764 views

Personalized content (authenticated with a cookie) and impact on SEO

We have a big website with a lot of content, that is doing pretty well in terms of SEO. Google likes the site, visitors like it as well. Our problem is that there is a lot of content that would only ...
3
votes
2answers
180 views

Google crawl rate is climbing but still no indexing?

This might belong on another stack site but idk. Soooo.... A year ago I started two sites that I didn't want indexed by Google or any other search engine since both sites were being developed based on ...
3
votes
3answers
92 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
3
votes
1answer
57 views

Structured Data Tool doesn't see JavaScript rendered content

Some structured data (namely price) of my page is rendered by JavaScript. According to fetching results in Webmaster tools Googlebot sees and renders it perfectly. However the structured data tool ...
3
votes
2answers
74 views

Increase in 5XX errors and 404 errors on GWT, are we under attack?

A lot of URLs are suddenly giving us 404s and 500s. On closer inspection, these URLs do not exist on our website and are being linked from random sources. As more URLs were inspected on GWT, a ...
3
votes
2answers
185 views

Does Google index AJAX websites after running the JavaScript?

As I understand Google depreciated its ajax crawling scheme they came up with back in 2009. I run single page app powered by Anguar frontend with push state navigation and my question is around ...
3
votes
1answer
72 views

Fetch as Google says my page is “Temporary Unreachable”

When looking into it in WebmasterTools, I saw that there is a "Fetch as Google - Temporarily Unreachable Issue". It is for an image, and font. How do I resolve this issue?
3
votes
1answer
87 views

Should I remove Google indexed pages before requesting a review after thin content manual action

Following on from my previous question - NOINDEX large number of pages to fix Google thin content manual action? Firstly, If I noindex all pages causing the manual action, is that enough for me to ...
3
votes
1answer
286 views

Webmaster Tools shows URL errors linked from http://com.example.www/:http

I have a bog standard WordPress 4.2.1 site. I recently added it to Webmaster Tools and it is showing me URL errors (404s) that are linked from a weird URL: http://com.example.www/:http (I have ...
3
votes
1answer
72 views

Is Google less likely to index URLs that don't match the directory structure of the page they are linked from?

Worded the question title the best I could. I have a website. The site has a category of articles - award winners. the URL to this category, a page with a list of winners, is: example.com/award-...
3
votes
1answer
75 views

Is there a way to stop a keyword from appearing in Google Webmaster Tools?

I added a web site a while ago to Google's index and it contains lots of links with the text "Permalink" that, well, link to permalinks of different content/pages. Now "Permalink" is the #1 keyword ...
3
votes
2answers
198 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
3
votes
1answer
674 views

Google crawlers screen size

Imagine a I have two types of images (A and AAA(heavier)) for the same website: depending on the size of the screen it will show one image or other. Now I'm wondering what images (A or AAA) loads ...
3
votes
2answers
100 views

Directing crawlers to content in language per language sub-domain

I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
3
votes
1answer
535 views

Webmaster Tools show the “Total Indexed” number to be zero

I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
2
votes
4answers
969 views

Google bots are severely affecting site performance

I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on ...
2
votes
3answers
587 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
2
votes
2answers
587 views

Is hiding content with JavaScript or “text-indent: -9999px” bad for SEO?

So apparently hiding content using "display: none" is bad for SEO and seen by googlebot as being deceptive. This according to a lot of the posts I read online and questions even on this site. But ...
2
votes
3answers
668 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
2
votes
3answers
134 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
2
votes
3answers
50 views

Strategy for mobile redirection and using alternative and canonical links

I had a separate mobile site uploaded so as to make Google happy. My big problem is that I had a desktop page that was once showing in the top 3 results on Google, and now it and its mobile ...
2
votes
1answer
749 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
2
votes
2answers
90 views

Will multilingual content on same page works to avoid content duplication

We have a website, which is having old/new vehicle deals for purchase. We are getting these data from third party website (legally) and all the pages are in English language. So, Here our issue is ...
2
votes
4answers
2k views

Does updating site too frequently hurt SEO rankings?

I recently updated my site, like 3-4 times in 2 months. At first everything was good, (i.e., Google was updating my pages' SERP content in 2-3 days). But now I did kind of a major change, I switched ...
2
votes
3answers
3k views

How much time does Googlebot take to crawl the page after submitting it to index

I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority. I've a page linking to newly added jobs ...
2
votes
3answers
188 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
2
votes
2answers
131 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
2
votes
2answers
93 views

Using table tag to structure a web page decrease referencing quality? [duplicate]

I am student in web programing. I use massively <table> ... </table> to structure my web page. My professor once told me it could decrease the referencing quality for my websites, so I ...
2
votes
2answers
202 views

What's the smallest page size for Googlebot to not go “Soft 404”?

Google appears to have two documents for Soft 404: https://support.google.com/webmasters/answer/181708?hl=en&ctx=tltp https://support.google.com/webmasters/answer/2409443?ctx=MCE&ctx=S4 ...
2
votes
1answer
58 views

Does Googlebot determine the value of a link by its position on the page?

It's well-known that the SEO value of a website basically depends on how much it has been referenced, linked, and shared. From Googlebot's point of view, is a link at the top of a page more valuable ...
2
votes
2answers
241 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
2
votes
3answers
1k views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
2
votes
1answer
329 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
2
votes
2answers
2k views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the /...
2
votes
1answer
47 views

How to present videos for Google Bot

i have a website which streams Flash Videos. For Google i have a special page due to using AJAX for normal Users (see articles on "_escaped_fragment_" for this). What i want to know is how i should ...
2
votes
3answers
166 views

Backlinks: How does a company's name as the anchor text help with SEO?

Often I see footer links like: SEO by COMPANY NAME How does this help with SEO other than helping a company rank for "Company Name"? When nearby words like "SEO by" get picked up by a search engine ...
2
votes
4answers
449 views

Google Webmaster Tools reporting crawl error 404 (Not found) for a page which has never existed

Google Webmaster Tools is reporting a crawl error "404 (Not Found)" for an .html page which has never existed on a web site I developed. The page being reported is "tyuiabmog.html", which doesn't ...
2
votes
2answers
758 views

Google webmaster Verification failed

I have a site created by Ruby on Rails. I had successfully verified the site with Google Webmaster Tools some months ago. However, recently Webmaster Tools has been asking me to verify my site. I ...
2
votes
2answers
41 views

When would Google crawler give up crawling a URL with 5xx error

How many times will Google crawler attempt to recrawl a URL that has a 5xx error occurring intermittently? When would it give up and drop the URL from the Index?
2
votes
2answers
54 views

Google crawler accessing non existing pages

I am looking Search Console and found Crawl errors for following URLs on my site: example.com/mobile example.com/m I never mentioned these pages anywhere but why Google is accessing it and what ...
2
votes
2answers
44 views

Will googlebot misinterpret my site if I had no spacing between anchor tags?

Some sections of my website involve displaying a list of photo thumbnails. The source code to that is similar to the following when condensed: <div> <a href="1">1</a> <a href="2"&...
2
votes
1answer
136 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
2
votes
2answers
2k views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
2
votes
2answers
351 views

Googlebot algorithm for crawling?

Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?
2
votes
1answer
677 views

Stop Google crawling links completely

I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told ...
2
votes
2answers
209 views

Get google to index JavaScript pages as separate pages

I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. ...