2

I'm planning to make a search engine, so I've downloaded this script. It works well, but I have to add a list of websites manually in a php file for the search engine to index. Is there any kind of database/file which lists a real lot of sites?

Improve this question

3 Answers 3

Reset to default
2

Probably the fastest way to pull together such a list is to scrape DMOZ and spider from there on out.

Improve this answer
2
  • I'll try that, but I am very bad at php (that's why I bought the script!)
    – Nick
    Jun 7, 2012 at 15:16
  • 3
    You don't have to scrape DMOZ, they provide updated archives of all the URL's in their directory.
    – Anagio
    Jun 8, 2012 at 10:46
2

This may help : Where can I download list of all .com domains registered in the world

Also as of last year there were only 70 million sites which Yahoo indexed.

So you get all the domains and then fetch their Alexa rankings and start indexing in ascending order of rankings.

Improve this answer
1

There are over 600M websites in the world, and that number is probably going up 1000s a day. I don't mean to be an ass, but a $14 script and any webserver you come up with isn't going to be able to be a remotely fast or effective search engine.

Use something like Google Custom Search to setup a a Google frontend just for your websites, or if you actually wanted to rival Google just give up now.

Improve this answer
3
  • I know what you mean, It's just for myself - I'll be happy to just have indexed 40,000 domains (or so!). The guy who made it has a database of domains, but most of them are romanian.
    – Nick
    Jun 7, 2012 at 13:24
  • Well someone should rival Google.. don't you think? And now since they blocked the referral data and Analytics systems will stop working there's a good reason to rival it.
    – Alexandru Trandafir Catalin2
    Nov 25, 2012 at 23:14
  • @AlexandruTrandafirCatalin Yeah, there's DDG, Bing, Yahoo etc - all funded by multi-millionaire companies/people. I agree there's a good reason to rival it, but it's a wholly unrealistic idea.
    – Dunhamzzz
    Nov 26, 2012 at 9:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.