3

Say I want to retrieve all the sites on the web that have a specific link on them. For example I want to know all the backlinks made to my blog, on other websites. There are services out there that do this: http://www.backlinkwatch.com/index.php - was wondering how they achieve this functionality.

Is crawling the entire internet the only option or are there third-party ways of doing this, say using Google.

Improve this question

5 Answers 5

Reset to default
8

Depending on what you want, you might indeed have to crawl the entire internet; but there is an easier way. If you are mostly interested in what links are actually clicked on to bring people to the page; storing referrer information and dealing with that will do plenty.

Improve this answer
1
3

To be 100% accurate, yes you'd have to crawl the entire net.

Both Google and Yahoo! have this functionality in their respective Analytics/Webmaster tools, but not even they are fully comprehensive.

How do they do it? Their core business relies on crawling (nearly) the entire net.

Improve this answer
2
  • Here's a Matt Cutts video talking about registering with Webmaster Tools for this, and also why just doing backlink queries from the Google front-end is limited. Note, though, that he still says Webmaster Tools doesn't show you every single link.
    – Su'
    Jun 22, 2011 at 18:42
  • Webmaster tools ignores many links from low ranked and rarely crawled websites.
    – UpTheCreek
    Jun 22, 2011 at 21:48
1

This doesn't have 100% of incoming links to your site, but this tool helps a lot - http://www.opensiteexplorer.org/

You can also find this information in google analytics on a page by page basis.

Improve this answer
0

Even the third parties you might use to get this data crawl as much of the Internet as they can, it can't be avoided. But yes, you can use third parties to offload the work from your servers.

Improve this answer
2
  • what's the third-party way? I would assume Google has some sort of syntax for this, just don't know which one exactly.
    – Luca Matteis
    Jun 22, 2011 at 14:23
  • I think that's a paid service, you'll have to contact them directly.
    – Blindy
    Jun 22, 2011 at 14:31
0

I suppose Googling for link:yourblog.com would be too easy?

It's entirely possible to crawl the 'Net, if that's what you needed to do—a saturated T4 line would take a few months to do it (you're only downloading the HTML, not images). But if a simple search is enough that's what I'd recommend.

Improve this answer
1
  • 1
    link: is an awful substitute. Google barely shows any backlinks for sites nowadays.
    – DisgruntledGoat
    Jun 23, 2011 at 11:07