I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing.
I currently have authentication set up using htpasswd, will this prevent site crawlers or robots from crawling my site?