I know how to find dead links which go to 404 pages. However, these days not many links actually go dead, but instead they end up going to a domain squatter. I realise this is a tall order, but is there any way to find out whether a website is actually a domain squatter without actually going to each site with my browser and examining it to see if there is a picture of a girl with a rucksack etc.?
Possible detection methods for parked pages/domains:
Find junk phrases
Do a case insensitive search for common generic junk phrases such as, "what you need, when you need it" and "your source for virtually anything!".
Find invitations to purchase
Look for text such as "inquire about this domain" and "this domain may be for sale".
Test for 404s on random subpages
Test for redirects on random subpages
Other parked domain systems redirect
Search for the domain name in meta tags
Several parked domain templates use the following format for the author meta tag:
Others put it in the description:
In each case, the domain is the only thing in the 'content' attribute. This is unlikely to be the case for active sites.
Look for the frameset tag
Some parked domain templates use the
Use multiple tests
No single one of these tests are necessarily reliable indicators of a parked domain on their own. You will likely have to combine multiple tests to create your own algorithm, then test and refine it based on a suite of known parked domains and known active ones.
There are things you can look for. Is the dominant element on the page an iFrame? Is the response a
I'd say it's very difficult, but that's at least some common characteristics.
There also seems to be a project on the Wikipedia