Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?

share|improve this question

2 Answers 2

up vote 8 down vote accepted

Can they see it? Yes. Can they find it? Not without help.

Web crawlers typically find pages to crawl by following links to them on other pages. Some crawlers (e.g. search engine crawlers) will also crawl pages listed in special XML files. So if there is no link to page on your website or any other website then that page will not be crawled (pages that contain the URL of that page but are in plain text will be found by Google).

However, once a page is found and crawled it may be crawled again even if all links to that page are removed from their respective websites. This is because pages that are crawled are then indexed (e.g. added to the crawlers list of pages to crawl again) so the crawler knows to crawl it again at a later time to look for changes. If you want to prevent this from happening you can do any of the following:

Most effective

  • Remove the page from the Internet
  • Changed the URL of that page (essentially removing the page and adding a new one)
  • Place it behind a login

Less effective

  • Block that page using a robots.txt file (which may be ignored)
  • Try to filter out bad bots by IP (which can change with every visit) or user-agent (may be spoofed)
share|improve this answer

Another way the page may be discovered is when you have links to other sites on that page.

The URL of you page will appear in their referrer logs, a nice time-pass of many webmasters is to briefly browse through those logs and see what others are saying about their pages.

Some sites seem to provide access to those logs without any access restriction, thus crawlers can reach them too...

To keep the page really secret, don't let it link out to external sites.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.