Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.
0
votes
1answer
14 views
How do search engines treat out-tracked links
Some websites have out tracking / outgoing click tracking so that when you click on a link such as mysite.com on example.com you momentarily get taken to example.com/outgoing/mysite.com which then ...
2
votes
2answers
37 views
Google SEO - Migrating website from a sub-directory of another website to its own domain name
At the present moment I have a website hosted on example.com/myWebsite, where example.com hosts in its root directory a different website. I have the domain example.net, which redirects to ...
2
votes
1answer
94 views
How frequently Googlebot fetch sitemaps? Is it depending on PageRank?
How much frequently Google fetches sitemaps?
I am now working with a high traffic website normally have 30 new posts per minute but currently it provides sitemaps which links to new 100 posts (3 ...
0
votes
1answer
45 views
Googlebot getting “301 Moved Permanently” according to “Fetch as Googlebot” in Webmaster Tools
I launched a website four days ago, and now when I try to crawl the website like Google bot in "Google webmaster tools" I end up with a response like this:
HTTP/1.1 301 Moved Permanently
Date: Thu, ...
0
votes
2answers
150 views
GWT: reporting crawling errors for non existing links
Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed.
They all mention ...
2
votes
1answer
42 views
“Google couldn't access your site because of a server connectivity issue”
In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site".
Under the Current Status it has:
DNS - SUCCESS
Server Connectivity ...
1
vote
1answer
173 views
Using rel=canonical and noindex in a 1-n partners enviroment
We sell a whole site (domain, etc) to partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original, but the one that is ...
2
votes
1answer
34 views
Which tools can help limit maximum page view per ip to limit scrapers and bots?
I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I Am thinking of going with Fail2ban and limiting hits from an ip maybe at around 1000 per day. ...
2
votes
1answer
53 views
Do Search Engine Spider's submit POST forms?
If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form?
For example, if I were to have 4 ...
2
votes
1answer
50 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
2
votes
1answer
40 views
Removing Google Webmaster URL Parameters
Some time before I have created a post for removing full site from Google Index, after following many advices I have removed large number of pages from Google Index but now facing another problem as ...
1
vote
3answers
43 views
Googlebot is blocked from my domain
My robots.txt is:
User-agent: *
Disallow: /
Sitemap: http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz
It has been the same for well over a week now, but when I go to ...
2
votes
2answers
101 views
How to remove Ever crawled web Pages by Google
I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
1
vote
0answers
29 views
Does Google bot always treat dot as word separator?
One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
1answer
22 views
Directing crawlers to content in language per language sub-domain
I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
1
vote
1answer
52 views
Googlebot craw rate is too slow for a huge site
I have a new website with 140k pages, according to my sitemaps.
After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
1
vote
1answer
46 views
GoogleBot can't reach my site
For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
3
votes
4answers
318 views
Google Bots Crawling my Site daily is this good or a bad thing?
Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
1
vote
1answer
3k views
How to fix Google 404 not found Crawl Errors?
I was checking on Google webmater tool for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
0
votes
2answers
53 views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
0answers
916 views
Is feeding Googlebot and Bingbot crawlers with “special” data common thing in RIAs?
So I faced next problem: my server and my RIA client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull no js crawler will see not ...
2
votes
2answers
156 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
2
votes
2answers
162 views
Would a 403 page, a blank page and a page redirect be indexed by search robots?
I am wondering if a disallow directive needs to be issued to robots on pages that will generate:
a 403 forbidden error;
a blank page, or;
a page redirect (to log in page for example).
Would not ...
-2
votes
2answers
89 views
How do I prevent duplicate indexing by Googlebot-Mobile?
I have only one version of my web-site, and it is optimised for both the desktop and the smartphone.
I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with ...
8
votes
3answers
2k views
Does Google cache robots.txt?
I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
1
vote
1answer
105 views
How to make image only website friendly to googlebot? [duplicate]
so I making this portfolio site for some artist. She draw all the menus and texts manually(on paper), then scanned, some "post processing" in photoshop and that's how the whole layout was created.
...
3
votes
5answers
258 views
SEO problems with multilanguage site [duplicate]
Possible Duplicate:
How can I get search engines to crawl my site and see a localised view of my data?
I have recently started a CMS project in PHP, the site will be multilingual. ...
0
votes
1answer
208 views
Google AdSense bot's algorithm and behavior [closed]
I am interesting in Google AdSense bot's algorithm and behavior with web site. I did not work with AdSense and I do not have account. So I need your help to understand:
1) Google's bot from time to ...
2
votes
1answer
274 views
Googlebot can't access my site when crawling from rootdomain
I can't explain why I get this message for my rootdomain result in Google :
trocmalin.com/
A description for this result is not available because of this site's robots.txt – learn more.
...
1
vote
3answers
171 views
Why isnt google crawling on my blog? [duplicate]
I have a blog.I made it on August,2010,so it has been a while but I am unable to make Google crawl on my site.I have requested Google to crawl but I still there is no response.Any reasons to note why ...
4
votes
3answers
555 views
nofollow doesn't stop crawling?
In my website, most of the pages has a linkedin sharing button written in this way:
<li>
<a id="share_linkedin"
...
2
votes
1answer
439 views
Prevent crawling of mobile site
I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users.
The mobile content is in /mobile/
The regular site will redirect mobile users to the ...
3
votes
2answers
71 views
Is offering Google Bot a lite version of site skin considered cloaking?
If you have multiple templates a user can choose from on a site (like a forum), and you offer Google Bot a "lite" skin, is this considered cloaking? Or is this considered okay?
The actual content ...
1
vote
2answers
130 views
Google Webmaster Tools complains about missing robots.txt
Nine days ago, I got a message Google Webmaster Tools:
Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt.
Well, but I don't have a robots.txt on ...
3
votes
2answers
111 views
Site re-styling affecting google ranking
I have to re-style a website in Drupal, and the client has a clear request: The site cannot drop its position on Google.
If I build a new Drupal site, with the same URLs, the same meta and title ...
3
votes
2answers
139 views
How can I prevent Googlebot from indexing web service URL's?
I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
2
votes
1answer
490 views
How to tell google to stop crawling my website via IP?
After a database server migration, Ive noticed that GoogleBot started getting errors when trying to crawl my site.
The reason seems to be that its hitting my site via IP address (and due to my config ...
2
votes
2answers
104 views
Get google to index JavaScript pages as separate pages
I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. ...
1
vote
0answers
117 views
Site returning 404 header to google, not sure why
A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by google at all (which is the main issue but I ...
2
votes
1answer
57 views
multilingual domains both crawled and come up to the wrong audience
I have a domain with multilingual support through subdomains. Each language version is identical besides the titles which are translated. e.g.:
domain.com
ja.domain.com
es.domain.com
I'm ...
1
vote
2answers
100 views
I don't think my site is being detected/crawled properly/having backlinks, and it's ruining SEO [closed]
This is really frustrating. I'm setting up a computer repairs business called Darley Tech. I have listed my site on many online directories to encourage backlinks. I have relevant and useful ...
1
vote
2answers
135 views
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.
I decreased crawl rate from Google webmaster tools.
But I'm supposing to ...
0
votes
0answers
17 views
hidden content and Googlebot [duplicate]
Possible Duplicate:
SEO - Hidden content before main site content
I have a website with the same content four times. Three of these four contents are hidden.
So, my question is: does the ...
0
votes
1answer
57 views
How do I get Google to crawl my content when it's only displayed when you fill in a form?
I have a webpage. It has a form and the "results" section is blank. When the user searches for items, and a list that pops up, he/she chooses one option from list and then the corresponding results ...
1
vote
1answer
116 views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
1
vote
1answer
409 views
Googlebot visit but no cache update - why?
I have made a new plain vanilla HTML website. I have been making regular modifications to it on an almost daily basis. The site is hosted by hostmonster and as part of their service they offer ...
1
vote
2answers
195 views
How can I tell GoogleBot that a subdirectory is now a subdomain?
I had about a million pages of a catalog indexed under a subdirectory, and now that's moved to a subdomain. GoogleBot is crawling each one of them and getting a 301 redirect to the new location. Even ...
1
vote
2answers
631 views
How to resolve “Google can't find your site's robots.txt” error?
I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
0
votes
0answers
17 views
Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]
Possible Duplicate:
How do spambots work?
I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
2
votes
4answers
238 views
Google bots are severely affecting site performance
I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on ...
