3

Say I want to retrieve all the sites on the web that have a specific link on them. For example I want to know all the backlinks made to my blog, on other websites. There are services out there that do this: http://www.backlinkwatch.com/index.php - was wondering how they achieve this functionality.

Is crawling the entire internet the only option or are there third-party ways of doing this, say using Google.

8

Depending on what you want, you might indeed have to crawl the entire internet; but there is an easier way. If you are mostly interested in what links are actually clicked on to bring people to the page; storing referrer information and dealing with that will do plenty.

3

To be 100% accurate, yes you'd have to crawl the entire net.

Both Google and Yahoo! have this functionality in their respective Analytics/Webmaster tools, but not even they are fully comprehensive.

How do they do it? Their core business relies on crawling (nearly) the entire net.

  • Here's a Matt Cutts video talking about registering with Webmaster Tools for this, and also why just doing backlink queries from the Google front-end is limited. Note, though, that he still says Webmaster Tools doesn't show you every single link. – Su' Jun 22 '11 at 18:42
  • Webmaster tools ignores many links from low ranked and rarely crawled websites. – UpTheCreek Jun 22 '11 at 21:48
1

This doesn't have 100% of incoming links to your site, but this tool helps a lot - http://www.opensiteexplorer.org/

You can also find this information in google analytics on a page by page basis.

0

Even the third parties you might use to get this data crawl as much of the Internet as they can, it can't be avoided. But yes, you can use third parties to offload the work from your servers.

  • what's the third-party way? I would assume Google has some sort of syntax for this, just don't know which one exactly. – Luca Matteis Jun 22 '11 at 14:23
  • I think that's a paid service, you'll have to contact them directly. – Blindy Jun 22 '11 at 14:31
0

I suppose Googling for link:yourblog.com would be too easy?

It's entirely possible to crawl the 'Net, if that's what you needed to do—a saturated T4 line would take a few months to do it (you're only downloading the HTML, not images). But if a simple search is enough that's what I'd recommend.

  • 1
    link: is an awful substitute. Google barely shows any backlinks for sites nowadays. – DisgruntledGoat Jun 23 '11 at 11:07