Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Say I want to retrieve all the sites on the web that have a specific link on them. For example I want to know all the backlinks made to my blog, on other websites. There are services out there that do this: http://www.backlinkwatch.com/index.php - was wondering how they achieve this functionality.

Is crawling the entire internet the only option or are there third-party ways of doing this, say using Google.

share|improve this question

migrated from stackoverflow.com Jun 22 '11 at 18:04

This question came from our site for professional and enthusiast programmers.

Depending on what you want, you might indeed have to crawl the entire internet; but there is an easier way. If you are mostly interested in what links are actually clicked on to bring people to the page; storing referrer information and dealing with that will do plenty.

share|improve this answer
Pingbacks can notify of backlinks, though those are only supported by select systems. secure.wikimedia.org/wikipedia/en/wiki/Pingback – Mufasa Jun 23 '11 at 22:29

To be 100% accurate, yes you'd have to crawl the entire net.

Both Google and Yahoo! have this functionality in their respective Analytics/Webmaster tools, but not even they are fully comprehensive.

How do they do it? Their core business relies on crawling (nearly) the entire net.

share|improve this answer
Here's a Matt Cutts video talking about registering with Webmaster Tools for this, and also why just doing backlink queries from the Google front-end is limited. Note, though, that he still says Webmaster Tools doesn't show you every single link. – Su' Jun 22 '11 at 18:42
Webmaster tools ignores many links from low ranked and rarely crawled websites. – UpTheCreek Jun 22 '11 at 21:48

This doesn't have 100% of incoming links to your site, but this tool helps a lot - http://www.opensiteexplorer.org/

You can also find this information in google analytics on a page by page basis.

share|improve this answer

Even the third parties you might use to get this data crawl as much of the Internet as they can, it can't be avoided. But yes, you can use third parties to offload the work from your servers.

share|improve this answer
what's the third-party way? I would assume Google has some sort of syntax for this, just don't know which one exactly. – Luca Matteis Jun 22 '11 at 14:23
I think that's a paid service, you'll have to contact them directly. – Blindy Jun 22 '11 at 14:31

I suppose Googling for link:yourblog.com would be too easy?

It's entirely possible to crawl the 'Net, if that's what you needed to do—a saturated T4 line would take a few months to do it (you're only downloading the HTML, not images). But if a simple search is enough that's what I'd recommend.

share|improve this answer
link: is an awful substitute. Google barely shows any backlinks for sites nowadays. – DisgruntledGoat Jun 23 '11 at 11:07

protected by Stephen Ostermiller Mar 6 at 2:13

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?