Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.
2
votes
2answers
162 views
Would a 403 page, a blank page and a page redirect be indexed by search robots?
I am wondering if a disallow directive needs to be issued to robots on pages that will generate:
a 403 forbidden error;
a blank page, or;
a page redirect (to log in page for example).
Would not ...
2
votes
0answers
916 views
Is feeding Googlebot and Bingbot crawlers with “special” data common thing in RIAs?
So I faced next problem: my server and my RIA client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull no js crawler will see not ...
1
vote
2answers
136 views
When will my old page stop appearing on Google? [duplicate]
Possible Duplicate:
Old pages still appearing in Google?
I recently bought a new address for my Blogger blog, from yannbane.blogspot.com to www.yannbane.com. However, www.yannbane.com ...
1
vote
2answers
216 views
sitemap and crawler
I am still newbie in the entire world of SEO. Let's take a look at my example. I have a website very similar to blog style site. There few patterns for urls, all of the are very simple and allowed to ...
1
vote
2answers
291 views
Is hiding content with JavaScript or “text-indent: -9999px” bad for SEO?
So apparently hiding content using "display: none" is bad for SEO and seen by googlebot as being deceptive. This according to a lot of the posts I read online and questions even on this site.
But ...
1
vote
3answers
171 views
Why isnt google crawling on my blog? [duplicate]
I have a blog.I made it on August,2010,so it has been a while but I am unable to make Google crawl on my site.I have requested Google to crawl but I still there is no response.Any reasons to note why ...
1
vote
1answer
116 views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
1
vote
1answer
206 views
hide some area (sidebar/footer) from googlebot. causing any SEO problem?
For example, I have a product page
http://www.myshop.com/product/iphone.html
in this page, normal users will see:
content area (about the iPhone itself)
sidebar links (links to other products: ...
1
vote
2answers
100 views
I don't think my site is being detected/crawled properly/having backlinks, and it's ruining SEO [closed]
This is really frustrating. I'm setting up a computer repairs business called Darley Tech. I have listed my site on many online directories to encourage backlinks. I have relevant and useful ...
1
vote
1answer
110 views
Do or can robots cause considerable performance issues?
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Our setup:
Lets say I have site ...
1
vote
2answers
61 views
Is using Progressive enhanment, over riding <a> tags with .js considered cloaking and can it be see by search engines
Im using .load to load in content in from a page /about.html, so that search engines can see what is going on ive made the links that trigger the .load tags to the page /about.html but for usablility ...
1
vote
1answer
870 views
Why google isn't updating my site title in search results? [duplicate]
Possible Duplicate:
Google doesn't seem to update the description or title of my homepage
I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
1
vote
2answers
134 views
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.
I decreased crawl rate from Google webmaster tools.
But I'm supposing to ...
1
vote
1answer
100 views
Could this be considered duplicate content? [duplicate]
Possible Duplicate:
What is duplicate content and how can I avoid being penalized for it on my site?
I'm building a dating site at the moment. On the homepage I have a paginated list of all ...
1
vote
1answer
260 views
Should i use a 301 redirect? Is Googlebot case sensitive?
The Googlebot reports the following:
Pages with duplicate meta descriptions
/Default.aspx?p=2
/default.aspx?p=2
So my question is, should i use a 301 redirect from Default to default or vice ...
1
vote
1answer
3k views
How to fix Google 404 not found Crawl Errors?
I was checking on Google webmater tool for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
1
vote
1answer
525 views
“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists
In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow".
I did ...
1
vote
2answers
127 views
google webmastertools will curtom crawl rate affects indexing what are the effects of custom crawl rate
My site was running well for the past three years. Recently googlebot's request my server got increased. and at one time i got network unreachable error. When i contacted my hosting company they asked ...
1
vote
3answers
42 views
Googlebot is blocked from my domain
My robots.txt is:
User-agent: *
Disallow: /
Sitemap: http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz
It has been the same for well over a week now, but when I go to ...
1
vote
1answer
104 views
How to make image only website friendly to googlebot? [duplicate]
so I making this portfolio site for some artist. She draw all the menus and texts manually(on paper), then scanned, some "post processing" in photoshop and that's how the whole layout was created.
...
1
vote
2answers
127 views
Google Webmaster Tools complains about missing robots.txt
Nine days ago, I got a message Google Webmaster Tools:
Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt.
Well, but I don't have a robots.txt on ...
1
vote
2answers
621 views
How to resolve “Google can't find your site's robots.txt” error?
I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
1
vote
1answer
482 views
How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?
I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
1
vote
1answer
83 views
Bots and scripts [duplicate]
Possible Duplicate:
Will AJAX-fetched material be invisible to search engines?
Can bots understand what's going on in javascript or vbscript? Most of the social networking modules and ad ...
1
vote
1answer
409 views
Googlebot visit but no cache update - why?
I have made a new plain vanilla HTML website. I have been making regular modifications to it on an almost daily basis. The site is hosted by hostmonster and as part of their service they offer ...
1
vote
0answers
29 views
Does Google bot always treat dot as word separator?
One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
1answer
50 views
Googlebot craw rate is too slow for a huge site
I have a new website with 140k pages, according to my sitemaps.
After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
1
vote
1answer
46 views
GoogleBot can't reach my site
For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
1
vote
1answer
22 views
Directing crawlers to content in language per language sub-domain
I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
1
vote
2answers
195 views
How can I tell GoogleBot that a subdirectory is now a subdomain?
I had about a million pages of a catalog indexed under a subdirectory, and now that's moved to a subdomain. GoogleBot is crawling each one of them and getting a 301 redirect to the new location. Even ...
1
vote
0answers
117 views
Site returning 404 header to google, not sure why
A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by google at all (which is the main issue but I ...
1
vote
1answer
172 views
Using rel=canonical and noindex in a 1-n partners enviroment
We sell a whole site (domain, etc) to partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original, but the one that is ...
1
vote
2answers
118 views
Confused by Fetch as Googlebot behaviour
I have a site that I 'm trying to index and I am a bit confused by what I should expect to see in the result of the Fetch as Googlebot tool. The site uses hashbangs, so according to Google it should ...
1
vote
1answer
203 views
Google search : pubdate issue
At some point this year, Google Search started to show the publication date.
When I updated my website, I noticed that the publication date was actually my birthday, simply because it's the first ...
1
vote
0answers
72 views
“workaround=” in IIS6 logs
A certain page on our website keeps generating entries like the following on our IIS6 logs:
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707368055555 80 - 66.249.85.65 - ...
0
votes
1answer
134 views
Googlebot following invalid URLs. How do I troubleshoot the problem?
I'm getting errors generated by Googlebot while following invalid URLs. Checking with Google Webmaster tools confirmed that the problem was the bot:
I've checked the source code and the HTML ...
0
votes
2answers
106 views
Will robots.txt block crawling things like menu and footer links?
If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
0
votes
1answer
67 views
Facelets SEO advice
Before I deploy the new version of my site, I would like to know a little more about facelets and SEO.
My previous version of site was written in JSF and I noticed that when I submitted it to the ...
0
votes
1answer
43 views
Googlebot getting “301 Moved Permanently” according to “Fetch as Googlebot” in Webmaster Tools
I launched a website four days ago, and now when I try to crawl the website like Google bot in "Google webmaster tools" I end up with a response like this:
HTTP/1.1 301 Moved Permanently
Date: Thu, ...
0
votes
2answers
88 views
Hide elements for Google
I have some elements which I don't want Google to see. Usually repetitive content, e.g. header and sidebar.
As Googlebot doesn't handle AJAX requests, I load these resources with AJAX and insert them ...
0
votes
2answers
112 views
Anyone heard of 'tank tracking'?
Someone I know told me about some seriously cutting edge blackhat SEO that he called 'tank tracking'. He said that it is some sort of code (he believes written in Python) that 'sits' around the ...
0
votes
1answer
207 views
Google AdSense bot's algorithm and behavior [closed]
I am interesting in Google AdSense bot's algorithm and behavior with web site. I did not work with AdSense and I do not have account. So I need your help to understand:
1) Google's bot from time to ...
0
votes
1answer
14 views
How do search engines treat out-tracked links
Some websites have out tracking / outgoing click tracking so that when you click on a link such as mysite.com on example.com you momentarily get taken to example.com/outgoing/mysite.com which then ...
0
votes
2answers
53 views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress ...
0
votes
4answers
210 views
Hide your website links from search engne
My Website Link is posted in someone else website, i asked for removal but can't get reply, google crawl his website and my website link is posted on that how to control that is that is possible?
i ...
0
votes
2answers
352 views
When using Mod_rewrite does google bot see the original url or the rewritten url?
If i use the bellow code in my .htacess file would google bot (or any other search spiders) see the example.tumblr.com url or would they see the same url as the user ie. example.co.uk/blog ?
...
0
votes
1answer
216 views
How does Google feel about overflow: auto?
I am planning to add some keyword rich text to my homepage at the very top. I have written almost a mini essay and putting all this text at the very top left of my page wouldn't look too nice. I was ...
0
votes
1answer
80 views
Specifying crawlers to not crawl Links which are dependent on external API's
I have links on my site which lead to internal pages that depend on first bringing data from an external API. This only takes time the first time they are pushed. Links to pages which exists in the ...
0
votes
2answers
150 views
GWT: reporting crawling errors for non existing links
Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed.
They all mention ...
0
votes
0answers
17 views
hidden content and Googlebot [duplicate]
Possible Duplicate:
SEO - Hidden content before main site content
I have a website with the same content four times. Three of these four contents are hidden.
So, my question is: does the ...
