How can I detect links which point to domain squatters?

Question

I know how to find dead links which go to 404 pages. However, these days not many links actually go dead, but instead they end up going to a domain squatter. I realise this is a tall order, but is there any way to find out whether a website is actually a domain squatter without actually going to each site with my browser and examining it to see if there is a picture of a girl with a rucksack etc.?

Answer 1

Possible detection methods for parked pages/domains:

Find junk phrases

Do a case insensitive search for common generic junk phrases such as, "what you need, when you need it" and "your source for virtually anything!".

Find invitations to purchase

Look for text such as "inquire about this domain" and "this domain may be for sale".

Test for 404s on random subpages

Visit testdomain.com/randomstring. If you get a 404, or the page itself contains the text '404' or 'not found', it's probably not parked.

Test for redirects on random subpages

Other parked domain systems redirect testdomain.com/randomstring to testdomain.com.

Search for the domain name in meta tags

Several parked domain templates use the following format for the author meta tag:

<meta name="author" content="Nameofdomain.com" />

Others put it in the description:

<meta name="description" content="nameofdomain.com">

In each case, the domain is the only thing in the 'content' attribute. This is unlikely to be the case for active sites.

Look for the frameset tag

Some parked domain templates use the <frameset> tag with multiple internal frames to pull in external content (often from 'information.com'), but otherwise feature nothing else on the page.

Use multiple tests

No single one of these tests are necessarily reliable indicators of a parked domain on their own. You will likely have to combine multiple tests to create your own algorithm, then test and refine it based on a suite of known parked domains and known active ones.

Answer 2

There are things you can look for. Is the dominant element on the page an iFrame? Is the response a 301/302 that takes you off-domain? (many squatters will simply 302 or 301 you to their landing page). Is the link/text ratio incredibly high?

I'd say it's very difficult, but that's at least some common characteristics.

There also seems to be a project on the Wikipedia linkrot page referring to some project that attempts to do this: http://en.wikipedia.org/wiki/Wikipedia_talk:Linkrot - details are sketchy though.