I just learned that we can be penalized for letting Google crawl and index our database search results.
Question: If they don't want to list Yellow Page type results, then why do they? These sites have been around forever, and the domains haven't changed. I just did a search for specific keywords on a friend's website. He ranks #7. The first six are all search results pages for well known Yellow Page type sites.
I'm a back-end developer so this all new to me. I reviewed the Webmaster Guidelines and saw this (emphasis mine):
Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines.
That is very subjective.
- My client has a page listing the names of all companies in our database.
- Each company name is a link to the search result page for that exact phrase.
- That result page in turn links to their company profiles in the different publications.
- The same company might have a profile in more than one publication.
- The profiles may be similar, but would list different product categories depending on the publication.
This was originally set up because the client was trying to compete with Yellow Pages and the like. And we are ranking pretty well when people search for particular companies. But I don't want to get penalized.
We were linking to the search results rather than directly to the profile because one company might have multiple profiles.
However, the client now wants to segregate the different publications more. So, I can save a click for users if the company list links directly to the profile and skips the search results page.
Question: Is the list of company names and the profile for each company acceptable content for Google? Do you think we will be penalized for letting them crawl and index that?
We were just about to add a similar list of all categories in the database where each category would link to a list of the companies in that category. I think this has value for users coming from search engines. But it's subjective.
Question: Since it is dynamically generated, Google could randomly request words like Viagra and we would present a "No matches found." page. That page would have a noindex meta tag. But is this enough to get us penalized?
Note: The actual search form uses POST which I believe Google avoids. We would only generate links for exact category names that exist in the database. So we wouldn't be inviting them to crawl a search results page, but more a directory landing page. However, nothing is there to stop Google from hunting for content by manipulating the URLs.
Vent: I know some basic SEO, and I've always gone with the idea of thinking about our users first - provide them with the content they are looking for - and let Googlebot figure things out on its own. It seems counter-intuitive to me to have to tell Google to stop crawling my site. The same thing goes for nofollow on partnership links (which I also just learned about). IMO, Google should just figure out what is relevant/valuable and display that. They shouldn't penalize sites for having content that doesn't interest them.
Aside: If they don't want to crawl useless pages, why are they still requesting pages that have been sending 301, 404, or 410 for more than a year? And no, there are no inbound links to these pages.