1

I have a site that allows people to look up words that "Start with" or "End with" a certain set of characters. I am trying to figure out how to get off on the right foot with search engines and I was wondering:

Is it better to have URLs that appear to be unique pages? For example:

mySite.com/StartsWith/pred

instead of 

mySite.com/Words.aspx?StartsWith=pred

Both of these would return the same data, but I am wondering if the first is better because it appears to be a unique page to a crawler?

At the end of the day the source code will only contain about 6 pages in it, but with all of the StartsWith/EndsWith letter sets, there are probably millions of possible combinations that people could get to. How would I (and should I) create a site map for pages that do not physically exist, but have unique URLs with unique content? Are there any other steps I should take to make sure that crawlers can find all of these different combinations?

Update There will be no duplicate content on the site

Improve this question

2 Answers 2

Reset to default
1

Both of the URLs will appear to be unique pages to the search engines. Just because one has a query string doesn't mean the search engines don't see it as a new page.

You definitely want to use an XML sitemap to let the search engines know about your pages. You can also do things like have the most recently searched words appear on your home page to help spiders find and crawl those pages.

Improve this answer
3
  • Should I even attempt to create a sitemap that has every possible combination? It seems like it would have millions of entries. I've seen some sites that come up with pages in search results when you search for "Words that start with ***". Are they only able to do this because they have a site map that points to that page?
    – Abe Miessler
    Apr 27, 2012 at 15:11
  • I can't say for sure that's how the other sites do it but it is a good possibility. Sitemaps do support millions of pages although there is a special format for sitemaps that are very large. Fortunately that's easy to do so I wouldn't let that be a showstopper.
    – John Conde ♦
    Apr 27, 2012 at 15:13
  • Would that mean using a sitemap index with multiple sitemap files?
    – Abe Miessler
    Apr 27, 2012 at 17:28
1

Instead of generating sitemap.xml on a regular basis which can lead to complex file managment :

  1. Record each search on your site in your DB
  2. Each day get the 50 (depends on your website) most famous searches and ad theme in a dedicated "Trending searches" page, linked in the footer of your website.
  3. Set a reasonnable limit of links on this page (50 links seems OK)

A sitemap with millions of similar URIs (duplicate content) isnt useful.

Improve this answer
2
  • There is no duplicate content. I will add a note to the original question. Also, the site map would not change once it was created
    – Abe Miessler
    Apr 28, 2012 at 16:44
  • You will have some effectively duplicate content: I'm pretty sure the pages listing words starting with, say, "eith" and "eithe" will be pretty similar. Of course, you could fix this particular issue pretty easily with redirects or rel=canonical links (e.g. to the longest common prefix) if you wanted.
    – Ilmari Karonen
    Apr 29, 2012 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.