Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I know how to find dead links which go to 404 pages. However, these days, not many links actually go dead, but instead they end up going to a domain squatter.
Is there any way to find out whether a website is actually a domain squatter without actually going to each site with my browser and examining it to see if there is a picture of a girl with a rucksack or something similar?
Possible detection methods for parked pages/domains:
Do a case insensitive search for common generic junk phrases such as, "what you need, when you need it" and "your source for virtually anything!".
Look for text such as "inquire about this domain" and "this domain may be for sale".
Visit testdomain.example/randomstring. If you get a 404, or the page itself contains the text '404' or 'not found', it's probably not parked.
Other parked domain systems redirect testdomain.example/randomstring to testdomain.example.
Several parked domain templates use the following format for the author meta tag:
<meta name="author" content="Nameofdomain.example" />
Others put it in the description:
<meta name="description" content="nameofdomain.example">
In each case, the domain is the only thing in the 'content' attribute. This is unlikely to be the case for active sites.
Some parked domain templates use the <frameset> tag with multiple internal frames to pull in external content (often from information.com), but otherwise feature nothing else on the page.
No single one of these tests are necessarily reliable indicators of a parked domain on their own. You will likely have to combine multiple tests to create your own algorithm, then test and refine it based on a suite of known parked domains and known active ones.
There are things you can look for. Is the dominant element on the page an iFrame? Is the response a 301/302 that takes you off-domain? (many squatters will simply 302 or 301 you to their landing page). Is the link/text ratio incredibly high?
I'd say it's very difficult, but that's at least some common characteristics.
There also seems to be a project on the Wikipedia linkrot page referring to some project that attempts to do this: Wikipedia talk:Link rot - details are sketchy though.
