How to properly add lists of content for SEO

Question

A list essentially boils down to a query, which can be modeled as a SQL query. A SQL query has this form:

SELECT * FROM LIST
  WHERE a = x
    AND b = y
   SORT a
  LIMIT 10

The first part is the list, where you select attributes from. The next part are a set of conditions, which can be in any order. Then you can sort the results. And limit the results to a small or manageable chunk or subset.

You can represent this in a URL using query parameters. So it could be:

 /?a=x&b=y&sort=a&limit=10

In fact, it can be in any order, so these are exactly the same:

 /?b=y&a=x&sort=a&limit=10
 /?limit=10&b=y&a=x&sort=a

This means different URLs give the same content. That leads to problems with SEO as described here. The main problem is that you want search engines to not see duplicate content, or content that is only slightly varied (as in category filtering vs. most recent sorting, etc.).

But my question is then, how you model/organize lists so that they get an SEO advantage. For example, I have seen pages like this:

• List of words (paginated).
• List of words starting with letter "a".
• List of 3-letter words starting with letter "a".
• List of fruits starting with "a".
• List of companies starting with "a".
• etc.

All of these are just words starting with the letter "a", just different subsets. So it seems that to a search engine, there is one "master" list of data, and all these "list" URLs are simply junk. But I don't see how it would know that.

Even still, I think these lists are valid. For example, knowing all the 3-letter words starting with "a" is great for Scrabble the game. Knowing which companies start with "a" is helpful to lookup a phone number for a company you barely remember. Etc.. These lists, whose content is exactly the same, just different subsets, are useful from a customer perspective.

But then you go slightly further and say "sort the list by x", and now the content is just in a different order. I don't think this counts as valid/valuable content. It's different than a special subset.

So my question is, how I go about creating these lists so search engines find them as "valuable" additions of information, rather than discarding them as duplicate content, or lowering their impact because they seem too closely related.

Answer 1

"limit 10" is way too small when talking about word lists. It should never be anything less than "limit 1000". No user is going to want to click "next" 100 times just to see the whole list. 1000 words, or even 10,000 words will easily fit in a reasonably sized page.

Most of your problems can be eliminated by increasing the number of words per page. Most of your examples should be just one page long.

You shouldn't allow search engines to index beyond page 1 in pagination, or various ways of sorting the same list. You can use robots.txt to disallow crawling of certain parameters, or use meta noindex tags on those pages. This is because it just isn't valuable to have users be able to land on "page 2 of companies starting with a, sorted by length".

Your URLs should not correspond quite so directly to your database query for SEO. Rather than query parameters for a=x&b=y you should describe what those accomplish in the URL. A much better URL would be /company-names-starting-with-a. Sort and pagination can handled with URL parameters, because you are going to prevent those parameters from being crawled and indexed. So /company-names-starting-with-a?sort=length&page=2 might be a valid URL, but it wouldn't be indexable by search engines.

Ideally you would write a paragraph of text for each list you produce saying what it is useful for. That gives you the opportunity to get other keywords onto your pages. "This list of 3 letter words starting with A is useful for Scrabble." will allow people that are searching for Scrabble word lists to find your site.

You might want to prevent the portion of the page containing the words in your word lists from getting indexed. The words in the word list themselves aren't really indexable content or search engine friendly. You aren't targeting people who search for "all any awe", but rather people who search for "three letter words starting with a". One way of doing that is using AJAX. Your HTML page wouldn't contain any of the words itself. It would make an AJAX call to a separate URL that would return the word list. That separate URL would be blocked by robots.txt.

Even if you do all this as well as possible, search engines still may not want to index your pages. There are already sites out there that provide word lists like this. You should try to find ways to make your site better than the competition. Just having word lists is probably not sufficient. Your site should be some combination of:

• Faster
• More usable
• More comprehensive
• More authoritative
• More accessible
• Better features
• Unique
• Less spammy

I know that a few years ago Google started cracking down on sites that created pages for phone numbers. There are only so many of a certain type of site that Google needs before their results start to look like garbage. Especially for pages that are automatically created by slicing a database, you need to up your game compared to what is already available.