Knowing your online Competition how to reveal 301’s directing domains

Question

I truly believe that any webmaster entering any niche should investigate their online competition beforehand to know what they're up against and to see if their goals are realistic and affordable in both time and money.

As you may know, there are a few ways of investigating your competition online. Rankings, local rankings, citations, social mentions and of course backlinks. This question is not related to any of the above but is related somewhat because it has the same end goal which you'll see momentarily.

So hopefully my question and the answers to follow may aid some other webmaster encountering the same problem. I'll illustrate the problem as a scenario that'll hopefully give you an idea of what I believe and what I'm asking.

The Scenario

Your website has been ranking top for almost 2 years within a fairly competitive niche. Then, out of nowhere, you drop in ranking and a site that has never been on page 10, never mind position 10, is all of a sudden in top.

An immediate reaction might be foul play and you investigate their content and backlinks, only to find that their content is extremely thin and they have zero backlinks. So the possibility here is that the backlinks have not indexed for that site with whatever tool you use to check for these yet. Or, Google for whatever reason has made an error that will resolve itself.

4 months on and the same thin site with 0 backlinks (Only those from expected bot/scrapers - webstats etc) is still top in rankings.

So it looks like this site is here to stay and you start thinking about other reasons to why your site is being outranked.

One possible reason could be that your own rankings has slipped, along with many other sites but having all those sites drop in ranking is unrealistic so the only way forward would be to believe that they have earned their rankings through either reputation with Google or foul play. The conclusion here would be - or least mine - that the site in question has recently moved from one domain to another. They may just have a simple 301 re-direct from one domain to another, or they may have told Google within Google Webmaster Tools they have moved site. Scenario over.

The Question

My question is fairly simple and I apologise for the long paragraphs of text but I want a good idea of what is trying to be achieved here. Enough typing, here's the question!

Is it possible to find out if one domain has moved from one domain to another, either using 301 re-directs or moving in Google Webmaster Tools?

Answer 1

Knowing the 301 redirected URL is possible in several cases but not all the time.

Several online tools like Ahrefs or MajesticSEO can display this kind of information for you. You just need to look at backlinks classified under "Redirect" status.

Example for Ahrefs:

However, you need to know a good SEO professional don't want people find the 301 redirect because in most of the cases, the redirected URL received in the past very good backlinks from good SEO places (websites with high PageRank for example).
That's why in general, in order to hide these good SEO places (and thus maybe good opportunities for everyone to get very good backlinks), you can't find the redirected URL with online tools. Indeed, the professional has blocked for the redirected URL all robots (especially those from Ahrefs, MajesticSEO...) except those from Google.
As a consequence, the 301 redirect is powerful for SEO and hidden for people.

P.S.: cloaking can also be applied by the professional to hide backlinks to the redirected URL for visitors. Therefore, these good places for SEO are even better hidden.

Answer 2

There are several sites that will tell you all sorts of information about sites such as:

• Number of pages on the site
• Traffic estimates for the site
• Other sites hosted at the same IP address (almost what you want)
• Other sites registered to the same owner (again, almost what you want)

I could not find one that tells you what other domain names redirect to a site. It probably would not be a huge stretch for one of those sites to add this type of functionality.

If I were to try build a tool to answer this question, here is what I would do:

• Download the entire list of domain names that have DNS entries. This in itself is a project. You have to apply to each registry for access or pay a broker that has done that legwork. This question on stackoverflow has answers for both approaches
• Download a page from each of the 150 million domain names to see if it redirects to some other site. I figure you should be able to do about 50 per second and the crawl would take a little over a month. This task is shardable, so it could be sped up by having multiple machines working on it. The bandwidth and hosting costs for this crawler could end up being sizable, the bandwidth for 50 sites per second would probably be about 20mb/s assuming an average download of 50KB pre site.

Answer 3

I'd suspect this is more effective using the cross-domain canonical tag. Using the canonical tag would keep the non-indexed site active so you could continue building links that pass on to the money site (It's hard to send a link request for a page that has already been 301'd).

There aren't any automated/off the shelf ways that I'm aware of to detect 301s. And I don't believe there are any ways to be confident you found them all. A few options that might help you find some are:

DNS Lookup to see if you can find other domains by the same owner. This will give you a good starting place if you find one. There are some websites that are used to try and find networks of sites (spyonweb.com I believe is one) they use AdSense ID, affiliate ID, DNS info etc. In this case I wouldn't think the owner would leave any of these footprints but you never know.

Open source web crawler, should be able to be tweaked to only follow links that 301 and return results for your desired domain name.

Look for link sellers in your niche. This is probably being done for the sake of creating a buffer between the money site and possible bad links (it's much easier to shut off a single redirected site then it is to get a few hundred webmasters to remove links to your site if you get penalized). Could mean they are participating in link buys, using something like Xenu on the sellers site would turn up any 301s.

inanchor: search on Google. If there is an identifying brand that it ranks well for you might turn up some links that use the brand in the anchor text but go to a different domain (which then hopefully redirects to the domain you want).

If you happen to find 1 or 2 sites using these methods you'll likely turn up a pattern that can be used to find more.

All these would be moot however, if they are using the canonical tag instead of 301. I'm sure there are some additional creative searches you could do but these should give you a good start.

-Good luck

Answer 4

If you want to know what 301 redirects are in place currently, yes that is absolutley possible with tracert on the command line. It should show you each hop, you may have to look up a couple of IP addresses, but it will show you the 301 redirects when the URL is requested from the server.

I don't know if there is a tool that would record the 301 itself. You might have some luck with Wayback Machine, but I would not be too hopeful.

Historical SERPs and backlink checkers should have all the relevant domains that would be redirected.

Here is a web version I found, used one of my bitly links to show how it works. Tracks two 301s to my domain. https://wheregoes.com/trace/2021789857/