Crawling is a process associated with the actions of Bots. Search engine bots or scrapers will generally 'Crawl' a website to obtain data.
4
votes
1answer
130 views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
1
vote
3answers
171 views
Why isnt google crawling on my blog? [duplicate]
I have a blog.I made it on August,2010,so it has been a while but I am unable to make Google crawl on my site.I have requested Google to crawl but I still there is no response.Any reasons to note why ...
11
votes
4answers
421 views
Status of Crawlable Ajax?
I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang).
http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html
...
4
votes
2answers
203 views
How does Google find a domain with no links to it?
I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else.
I just discovered tonight that the page is already indexed in ...
