4

Does anyone know of a reliable expired domains database that I download and query against, without needing to connect through an API?

Also, something that has older data would be nice.

If such a database does not exist, what data sources do you recommend to setup my own.

1

2 Answers 2

3

http://www.olddomains.net/ is free, and http://www.dnware.com/products/database/expired.php is premium. It may be well worth your money to invest in the latter. They even provide sample database files.

2
  • I came across those 2 in my search, but had concerns with both. I believe olddomains.net doesn't let you download the database, but only query via http against their server. This wouldn't work for me. Also, they're a very very small database. The second one, dnware, all their data is in text, and it would take me a long time to parse and upload to a mysql database. I actually have some txt files from various sources, but the biggest problem is transfer time from txt to mysql. That's why I was looking for a ready database to query against. +1 for the help though.
    – sameold
    Commented Aug 24, 2012 at 19:00
  • @sameold - I might be able to write a script to parse the text files into MySQL format. Contact me if you'd consider that. (Click my username, homepage, contact form.)
    – ionFish
    Commented Aug 24, 2012 at 19:09
2

Scraping of the Chinese expired domain trackers

It is not pretty, but I've found the following trackers to contain valuable data:

Each year contains about 20 to 30 million expired domains.

I'm uploading my combined scraping of those websites to:

Both of those sites contain the same data, except that each one has some URLs randomly missing, so they can complement one another relatively well.

Their first data point is 2011-07-29 as far as I could find, and the data seems to continue to the present.

Both are highly IP throttled, but can be scraped in about a week with the following script:

#!/usr/bin/env bash
set -eu
dir=tmp/hupo
mkdir -p "$dir"
d="$(ls "$dir" | tail -n1)"
if [ -z "$d" ]; then
  # Smallest known working date:
  d=2011-07-29
else
  d="$(date -I -d "$d + 1 day")"
fi
while [ "$d" != 2023-01-01 ]; do
  echo $d
  # Last known with space:
  # http://static.hupo.com/expdomain_myadmin/2012-01-23(国际域名)%20.txt
  # Earlist known without space:
  # http://static.hupo.com/expdomain_myadmin/2012-02-03(国际域名).txt
  if [ "$d" \< 2012-02-03 ]; then
    space='%20'
  else
    space=''
  fi
  outfile="$dir/$d"
  set +e
  code="$(curl "http://static.hupo.com/expdomain_myadmin/$d(国际域名)${space}.txt" -o "$outfile" -w "%{http_code}")"
  stat=$?
  set -e
  if [ "$stat" -eq 0 ] && [ "$code" -eq 200 ]; then
    # Because of course they use CR LF, of course!
    # I kid you not ,some of them are invert sorted after some date.
    dos2unix "$outfile"
    sort -o "$outfile" "$outfile"
    echo sleep
    sleep 180
  else
    rm -f "$d"
  fi
  if [ "$stat" -eq 0 ]; then
    # Retry any potential network errors.
    d=$(date -I -d "$d + 1 day")
  fi
done

The webmasterhome.cn one is analogous.

I do not know their methodology, likely it is undocumented like most of this type of dataset, but I can give it the following characterization: it is not obtained through scraping alone. They must have had access to something like a zone file.

I know this because they contain several domain names which I'm very confident that the CIA used for their circa 2010 covert communication networks as I've documented at: https://cirosantilli.com/cia-2010-covert-communication-websites These are extremely likely therefore domains with no HTML backlinks to them, and must therefore have been obtained through other means.

I found these tracker websites by Googling for some of the CIA domains, and they were some the very few Google hits, among a few other non-Chinese trackers. Also, after scraping them, I was able to find other CIA websites, which further indicates that their data is real and valuable.

Further methodology and scripting described at: https://cirosantilli.com/cia-2010-covert-communication-websites/expired-domain-trackers

Not the answer you're looking for? Browse other questions tagged or ask your own question.