14

I truly believe that any webmaster entering any niche should investigate their online competition beforehand to know what they're up against and to see if their goals are realistic and affordable in both time and money.

As you may know, there are a few ways of investigating your competition online. Rankings, local rankings, citations, social mentions and of course backlinks. This question is not related to any of the above but is related somewhat because it has the same end goal which you'll see momentarily.

So hopefully my question and the answers to follow may aid some other webmaster encountering the same problem. I'll illustrate the problem as a scenario that'll hopefully give you an idea of what I believe and what I'm asking.

The Scenario

Your website has been ranking top for almost 2 years within a fairly competitive niche. Then, out of nowhere, you drop in ranking and a site that has never been on page 10, never mind position 10, is all of a sudden in top.

An immediate reaction might be foul play and you investigate their content and backlinks, only to find that their content is extremely thin and they have zero backlinks. So the possibility here is that the backlinks have not indexed for that site with whatever tool you use to check for these yet. Or, Google for whatever reason has made an error that will resolve itself.

4 months on and the same thin site with 0 backlinks (Only those from expected bot/scrapers - webstats etc) is still top in rankings.

So it looks like this site is here to stay and you start thinking about other reasons to why your site is being outranked.

One possible reason could be that your own rankings has slipped, along with many other sites but having all those sites drop in ranking is unrealistic so the only way forward would be to believe that they have earned their rankings through either reputation with Google or foul play. The conclusion here would be - or least mine - that the site in question has recently moved from one domain to another. They may just have a simple 301 re-direct from one domain to another, or they may have told Google within Google Webmaster Tools they have moved site. Scenario over.

The Question

My question is fairly simple and I apologise for the long paragraphs of text but I want a good idea of what is trying to be achieved here. Enough typing, here's the question!

Is it possible to find out if one domain has moved from one domain to another, either using 301 re-directs or moving in Google Webmaster Tools?

| improve this question | |
  • I would have thought the same tools that track backlinks could potentially track this information also. However, I have my doubt whether (many of) these tools actually follow links and therefore don't see the redirection (although that would also mean their backlink reporting wouldn't be entirely accurate). – MrWhite Aug 5 '13 at 20:45
  • +1. Great question. I assumed that when you move a site from within Webmaster Tools, that Google automatically adjusted the results from link:example.com in order to 'move the link juice', so to speak. This scenario is weird, to be quite frank. – Athoxx Aug 22 '13 at 9:47
  • I have an idea, could you share URL with no backlinks? – Zistoloen Aug 29 '13 at 10:47
  • 2
    @Zistoloen, then it would have a backlink, heh – Octopus Sep 11 '13 at 19:20
  • @Octopus: a nofollow backlink. – Zistoloen Sep 11 '13 at 20:53
3
+50

Knowing the 301 redirected URL is possible in several cases but not all the time.

Several online tools like Ahrefs or MajesticSEO can display this kind of information for you. You just need to look at backlinks classified under "Redirect" status.

Example for Ahrefs:

enter image description here

However, you need to know a good SEO professional don't want people find the 301 redirect because in most of the cases, the redirected URL received in the past very good backlinks from good SEO places (websites with high PageRank for example).
That's why in general, in order to hide these good SEO places (and thus maybe good opportunities for everyone to get very good backlinks), you can't find the redirected URL with online tools. Indeed, the professional has blocked for the redirected URL all robots (especially those from Ahrefs, MajesticSEO...) except those from Google.
As a consequence, the 301 redirect is powerful for SEO and hidden for people.

P.S.: cloaking can also be applied by the professional to hide backlinks to the redirected URL for visitors. Therefore, these good places for SEO are even better hidden.

| improve this answer | |
  • +1 As I can see this as a semi-working method. Depends if the redirect is detected. The problem is doesn't ahrefs and majestic use Google index? to discover? a redirecting domain won't be indexed, And I assume the ones shown are for sites with internal.com/?redirect=go:domain or something like that. – Simon Hayter♦ Sep 14 '13 at 12:42
  • Indeed, this is a semi-working method but I believe this is the best method we can find. – Zistoloen Sep 14 '13 at 14:01
  • Shame Google don't do a cache going back months to years ;\ – Simon Hayter♦ Sep 14 '13 at 14:09
  • Yes, Google is not perfect... – Zistoloen Sep 14 '13 at 14:23
  • I don't know about ahrefs but I am pretty sure majesticSEO has their own crawler and doesn't rely on Google for their index. – Joshak Oct 23 '13 at 16:44
2

There are several sites that will tell you all sorts of information about sites such as:

  • Number of pages on the site
  • Traffic estimates for the site
  • Other sites hosted at the same IP address (almost what you want)
  • Other sites registered to the same owner (again, almost what you want)

I could not find one that tells you what other domain names redirect to a site. It probably would not be a huge stretch for one of those sites to add this type of functionality.

If I were to try build a tool to answer this question, here is what I would do:

  • Download the entire list of domain names that have DNS entries. This in itself is a project. You have to apply to each registry for access or pay a broker that has done that legwork. This question on stackoverflow has answers for both approaches
  • Download a page from each of the 150 million domain names to see if it redirects to some other site. I figure you should be able to do about 50 per second and the crawl would take a little over a month. This task is shardable, so it could be sped up by having multiple machines working on it. The bandwidth and hosting costs for this crawler could end up being sizable, the bandwidth for 50 sites per second would probably be about 20mb/s assuming an average download of 50KB pre site.
| improve this answer | |
  • Nice try Stephen but you don't really give the answer. – Zistoloen Sep 12 '13 at 16:02
  • I'm hoping something better will come up, but after doing some research, I doubt it. – Stephen Ostermiller♦ Sep 12 '13 at 16:06
  • +1 for the DNS look up, through I had already done that, I checked who else was using the same name servers as well as the IP. Through I suspect the name servers are not the same and the IP is completely different. – Simon Hayter♦ Sep 12 '13 at 17:07
1

I'd suspect this is more effective using the cross-domain canonical tag. Using the canonical tag would keep the non-indexed site active so you could continue building links that pass on to the money site (It's hard to send a link request for a page that has already been 301'd).

There aren't any automated/off the shelf ways that I'm aware of to detect 301s. And I don't believe there are any ways to be confident you found them all. A few options that might help you find some are:

DNS Lookup to see if you can find other domains by the same owner. This will give you a good starting place if you find one. There are some websites that are used to try and find networks of sites (spyonweb.com I believe is one) they use AdSense ID, affiliate ID, DNS info etc. In this case I wouldn't think the owner would leave any of these footprints but you never know.

Open source web crawler, should be able to be tweaked to only follow links that 301 and return results for your desired domain name.

Look for link sellers in your niche. This is probably being done for the sake of creating a buffer between the money site and possible bad links (it's much easier to shut off a single redirected site then it is to get a few hundred webmasters to remove links to your site if you get penalized). Could mean they are participating in link buys, using something like Xenu on the sellers site would turn up any 301s.

inanchor: search on Google. If there is an identifying brand that it ranks well for you might turn up some links that use the brand in the anchor text but go to a different domain (which then hopefully redirects to the domain you want).

If you happen to find 1 or 2 sites using these methods you'll likely turn up a pattern that can be used to find more.

All these would be moot however, if they are using the canonical tag instead of 301. I'm sure there are some additional creative searches you could do but these should give you a good start.

-Good luck

| improve this answer | |
  • Digging through some reports in MajesticSEO it looks like their site explorer tool has redirects split out in their backlinks section. May give you what you need. – Joshak Sep 25 '13 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.