1

I know search engines first index the main page, then follow by its links and do the same with the pages by the links as the main page before. Ok, but what about folders and subfolders? Do search engines index them separately even if a folder is not mentioned in the source code of a page? For example I have the folder /some_folder and it's nowhere to be mentioned in the source code of pages, will it be indexed too?

If yes and I use static html pages and I have a page with URL like category/page (i.e. the page.html file in the /category folder), does it mean search engine will index this folder with html files and there will be two copies of the /category/page: one from indexing of the page after following by the link /category/page and one from direct indexing the /category folder with the same page as html file?

1

Search engines only index URLs that have unique content on them. If your folder URL doesn't return any useful content, search engines won't index it. That will be the case if the folder URL:

  • Returns a blank page
  • Has a error code such as 403 Forbidden or 404 Not Found
  • Redirects to some other URL

Search engines do try to discover content by crawling folders. If you use the URL /category/page on your site, search engines may very well try to crawl /category/ to see if they can find content there.

If you are powering your website with static files, directory URLs may have content that is automatically generated by your web server. Most web servers are configured to serve a directory listing. That is often enough "content" for search engines to index. Further more, search engines will follow the links in that directory listing and find all the other files you have in the directory.

If you don't want your directory listing indexed you can add an index.html to that folder to replace the directory listing with content of your own choosing. Alternately you can use Options -Indexes on Apache to turn off the generation of directory indexes and instead serve an error page.

Search engines never index the same URL multiple times, even if they find multiple ways to get to the same URL. If you link to /category/ and search engines also try find it because they expect it from /category/page they will still crawl and index that URL only once.

For directories there are some issues with URL canonicalization. All three of these URLs may return the same content:

  • /category
  • /category/
  • /category/index.html

Web servers usually redirect to add the slash to directory URLs. Even if you power the directory content with an index.html file you shouldn't use index.html in your links. Just link to the directory URL ending in the trailing slash.

  • Thank you! One more question. I know the size of a page matters for search engine and as I understand the total quantity of files or pages inside of a folder doesn't matter. Is that true? That is if a website has hundreds of thousands of pages or files, search engine still will index them regardless of their total quantity? – stckvrw May 13 at 11:28
  • 1
    Search engines will only index dozens or hundreds of pages from a brand new site. As a site gains reputation from other sites linking to it, search engines will be willing to index more and more files. Some sites get to the point where they have hundreds of thousands of pages indexed, but it takes a lot of quality content and a lot of recommendations of the site through links to get that. – Stephen Ostermiller May 13 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.