Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

0
votes
1answer
117 views

Mobile site redirection and Google crawlers

I have a HTML website where users can select a product (going through categories/subcategories) and then when they choose the product they can customize it in a Flash application. I would like to ...
0
votes
1answer
285 views

How to find first cached date and time of a website?

I found some of my webpage contents has been copied in other website. I need to check the cache date and time of both pages so that i can compare and guess which page is original and which one is ...
0
votes
2answers
400 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
0
votes
1answer
73 views

How do I get Google to crawl my content when it's only displayed when you fill in a form?

I have a webpage. It has a form and the "results" section is blank. When the user searches for items, and a list that pops up, he/she chooses one option from list and then the corresponding results ...
0
votes
0answers
17 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
-2
votes
2answers
267 views

How do I prevent duplicate indexing by Googlebot-Mobile?

I have only one version of my web-site, and it is optimised for both the desktop and the smartphone. I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with ...
-3
votes
2answers
147 views

Does Google bot crawl unused tags?

If a website a called foo.com has the following css, #LinkBuilder{ background:url("www.LoremEpsum.com"); } and the css id LinkBuilder is actually not getting used anywhere in the html of foo.com, ...