Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.
9
votes
2answers
244 views
How is Googlebot finding URLs that are only visible to authenticated users?
Here is one of my customers, performing some action after having logged in to his account. The unique token is simply an encrypted user id + timestamp.
94.254.xxx.xxx - - [02/Jul/2011:22:25:46 ...
9
votes
1answer
515 views
How did Google manage to crawl my 403 pages?
I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
8
votes
3answers
2k views
Does Google cache robots.txt?
I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
7
votes
1answer
332 views
Should I ping to Google sitemap-index.xml or only the sitemap file that has been modified?
I have a sitemap-index.xml with 7 files. After updating one of the 7 with new content should I ping Google with the index file or with the updated file?
In my index I also use lastmod so Google can ...
5
votes
3answers
423 views
Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]
Possible Duplicate:
Is it possible for web crawlers to see static pages without following a link to them?
I have some URLs (some pdfs and static HTML files) in my website that I want only ...
5
votes
2answers
136 views
Robots.txt and pattern matching
Adding this to my robots.txt
User-agent: *
Disallow: /*action=*$
How does robots not recognizing wild cards handle this?
5
votes
5answers
116 views
Which tools can help limit maximum page view per IP to limit scrapers and bots?
I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
5
votes
1answer
532 views
Am I covering the major search engine spiders in my anti-crawl protection white list?
I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
4
votes
1answer
293 views
Does htpasswd authentication prevent Google from crawling my site?
I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing.
I currently ...
4
votes
1answer
909 views
link rel=“alternate”, multiple languages and canonical urls
Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain.
On some ...
4
votes
2answers
689 views
Does duplicate content inside your own website matter for SEO? [duplicate]
Possible Duplicate:
What is duplicate content and how can I avoid being penalized for it on my site?
I'm not referring to the duplicate content where you copy content from another site and ...
4
votes
3answers
597 views
nofollow doesn't stop crawling?
In my website, most of the pages has a linkedin sharing button written in this way:
<li>
<a id="share_linkedin"
...
3
votes
5answers
392 views
How to make sure that Google indexes your site in less than a day?
I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
3
votes
2answers
114 views
Site re-styling affecting google ranking
I have to re-style a website in Drupal, and the client has a clear request: The site cannot drop its position on Google.
If I build a new Drupal site, with the same URLs, the same meta and title ...
3
votes
4answers
329 views
Google Bots Crawling my Site daily is this good or a bad thing?
Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
2answers
76 views
Is offering Google Bot a lite version of site skin considered cloaking?
If you have multiple templates a user can choose from on a site (like a forum), and you offer Google Bot a "lite" skin, is this considered cloaking? Or is this considered okay?
The actual content ...
3
votes
2answers
158 views
How can I prevent Googlebot from indexing web service URL's?
I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
3
votes
2answers
57 views
Preventing Adsense Crawling Specific Regions of the Site
In my website, I wan't adsense crawler to not consider certain parts of the HTML. Is there any snippet I can use. The reason is I am using a plugin from an external provider and as soon as I installed ...
3
votes
1answer
167 views
Google 404 errors on webmaster page
I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
3
votes
1answer
456 views
IIS Dynamic IP Restrictions Module and Googlebot
I consider using IIS Dynamic IP Restrictions Module in order to make it hard for DoS (Denial of Service) attacks. But I am not sure how that would work out for search engine bots like Googlebot.
Any ...
3
votes
5answers
269 views
SEO problems with multilanguage site [duplicate]
Possible Duplicate:
How can I get search engines to crawl my site and see a localised view of my data?
I have recently started a CMS project in PHP, the site will be multilingual. ...
3
votes
1answer
389 views
Webmaster Tools show the “Total Indexed” number to be zero
I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
3
votes
4answers
334 views
Personalized content (authenticated with a cookie) and impact on SEO
We have a big website with a lot of content, that is doing pretty well in terms of SEO. Google likes the site, visitors like it as well. Our problem is that there is a lot of content that would only ...
3
votes
2answers
53 views
Google SEO - Migrating website from a sub-directory of another website to its own domain name
At the present moment I have a website hosted on example.com/myWebsite, where example.com hosts in its root directory a different website. I have the domain example.net, which redirects to ...
2
votes
4answers
269 views
Google bots are severely affecting site performance
I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on ...
2
votes
1answer
68 views
Do Search Engine Spider's submit POST forms?
If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form?
For example, if I were to have 4 ...
2
votes
2answers
379 views
What does Google do with indexed pages returning 403?
I guess that Google removes already indexed pages that are now returning the HTTP error 403.
Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
2
votes
1answer
209 views
Stop bots from crawling old links with extensions
I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist.
So I'm trying to find out if there is a ...
2
votes
1answer
45 views
How to present videos for Google Bot
i have a website which streams Flash Videos. For Google i have a special page due to using AJAX for normal Users (see articles on "_escaped_fragment_" for this).
What i want to know is how i should ...
2
votes
2answers
365 views
Google webmaster Verification failed
I have a site created by Ruby on Rails. I had successfully verified the site with Google Webmaster Tools some months ago. However, recently Webmaster Tools has been asking me to verify my site. I ...
2
votes
2answers
110 views
Get google to index JavaScript pages as separate pages
I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. ...
2
votes
1answer
270 views
Making JSON page which renders on document ready indexable
I have a page which is pretty much 99% on document ready generated by various JS files.
It then gets and renders tons of JSON (I use tempo JS for rendering).
It all happens on document ready. So ...
2
votes
3answers
82 views
suspicious crawler activity
I'm noticing that I get accesses
66.249.66.198 - - [01/Jul/2011:17:13:46 +0200] "GET /img/clip.incubus.torrent.phtml HTTP/1.1" 404 143 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
2
votes
2answers
122 views
Backlinks: How does a company's name as the anchor text help with SEO?
Often I see footer links like:
SEO by COMPANY NAME
How does this help with SEO other than helping a company rank for "Company Name"? When nearby words like "SEO by" get picked up by a search engine ...
2
votes
3answers
304 views
Google Webmaster Tools reporting crawl error 404 (Not found) for a page which has never existed
Google Webmaster Tools is reporting a crawl error "404 (Not Found)" for an .html page which has never existed on a web site I developed. The page being reported is "tyuiabmog.html", which doesn't ...
2
votes
1answer
78 views
“Google couldn't access your site because of a server connectivity issue”
In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site".
Under the Current Status it has:
DNS - SUCCESS
Server Connectivity ...
2
votes
1answer
600 views
Google on capital letters in url
I created one of my websites with the following url structure:
http://domain.com/Animal/Pig
http://domain.com/Pigs
If I change the url map to lower case letters instead, will google reindex those ...
2
votes
2answers
81 views
Forbid content in google results summary
I put a professional directory project online since few weeks, Google has started to index some pages but i just noticed a flaw in the design of the pages.
Each page have an adress, some content and ...
2
votes
1answer
519 views
How to tell google to stop crawling my website via IP?
After a database server migration, Ive noticed that GoogleBot started getting errors when trying to crawl my site.
The reason seems to be that its hitting my site via IP address (and due to my config ...
2
votes
2answers
2k views
robots.txt for a CodeIgniter based website?
I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.
public_html
application
public
sql
style
system
design
images
...
2
votes
1answer
35 views
Can bots crawl through “checkbox” form elements which act as a filter for products?
For example, my site has a list of products, with a filter box which allows you to narrow by price, features, etc. My question is, can google crawl this form, see the url results (which is a bunch of ...
2
votes
1answer
59 views
GoogleBot can't reach my site
For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
1answer
59 views
multilingual domains both crawled and come up to the wrong audience
I have a domain with multilingual support through subdomains. Each language version is identical besides the titles which are translated. e.g.:
domain.com
ja.domain.com
es.domain.com
I'm ...
2
votes
2answers
158 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
2
votes
1answer
281 views
Googlebot can't access my site when crawling from rootdomain
I can't explain why I get this message for my rootdomain result in Google :
trocmalin.com/
A description for this result is not available because of this site's robots.txt – learn more.
...
2
votes
1answer
317 views
HTTP 303 redirection and robots.txt
On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
455 views
Prevent crawling of mobile site
I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users.
The mobile content is in /mobile/
The regular site will redirect mobile users to the ...
2
votes
1answer
86 views
Crawl Errors (404) Showing Up in WMT
I see from WMT that my site has 4 (404) crawl errors, each Linked from 2 separate pages. As this is a small directory listings site it would be difficult to delete the missing URLs from the db each ...
2
votes
1answer
67 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
2
votes
1answer
51 views
Removing Google Webmaster URL Parameters
Some time before I have created a post for removing full site from Google Index, after following many advices I have removed large number of pages from Google Index but now facing another problem as ...
