I'm planning to make a search engine, so I've downloaded this script. It works well, but I have to add a list of websites manually in a php file for the search engine to index. Is there any kind of database/file which lists a real lot of sites?


Probably the fastest way to pull together such a list is to scrape DMOZ and spider from there on out.

| improve this answer | |
  • I'll try that, but I am very bad at php (that's why I bought the script!) – Nick Jun 7 '12 at 15:16
  • 3
    You don't have to scrape DMOZ, they provide updated archives of all the URL's in their directory. – Anagio Jun 8 '12 at 10:46

This may help : Where can I download list of all .com domains registered in the world

Also as of last year there were only 70 million sites which Yahoo indexed.

So you get all the domains and then fetch their Alexa rankings and start indexing in ascending order of rankings.

| improve this answer | |

There are over 600M websites in the world, and that number is probably going up 1000s a day. I don't mean to be an ass, but a $14 script and any webserver you come up with isn't going to be able to be a remotely fast or effective search engine.

Use something like Google Custom Search to setup a a Google frontend just for your websites, or if you actually wanted to rival Google just give up now.

| improve this answer | |
  • I know what you mean, It's just for myself - I'll be happy to just have indexed 40,000 domains (or so!). The guy who made it has a database of domains, but most of them are romanian. – Nick Jun 7 '12 at 13:24
  • Well someone should rival Google.. don't you think? And now since they blocked the referral data and Analytics systems will stop working there's a good reason to rival it. – Alexandru Trandafir Catalin2 Nov 25 '12 at 23:14
  • @AlexandruTrandafirCatalin Yeah, there's DDG, Bing, Yahoo etc - all funded by multi-millionaire companies/people. I agree there's a good reason to rival it, but it's a wholly unrealistic idea. – Dunhamzzz Nov 26 '12 at 9:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.