0

So, imagine, we were blocking some URLs via robots.txt, but apparently these pages were getting backlinks.

Later, we decided to redirect these blocked pages to another domain of ours.

~1.5 month after we started redirection, our Google traffic dropped significantly.

Can backlinks to robots.txt blocked pages boost domain's ranking, thus hurting our domain's ranking possibilities and SE traffic to our domain after redirecting such pages?

1

To answer this, you have to know something of the mechanics behind how all of this works.

Every time a link is discovered, it is entered into a link table within he index. The source URL, the page with the link, is at least already within the URL table within the index. If the target URL, the page being linked to, is not in the index, it is a dangling link. This means that Google has not investigated both ends of the link yet and therefore no metrics can exist. When a link is discovered and entered into the link table, the target URL is entered into the fetch queue.

When the page is fetched, assuming nothing goes wrong, it is indexed and the link value metrics can be calculated and applied. This is done in a batch process periodically. It runs quickly and the last I heard this was run monthly, however, likely can run more often. The process is recursive. Let me give you an example. If a page has a particular PR, as links to that page are added and dropped, the PR for that page changes and therefore can effect the pages it links to. This is not a linear calculation and must be repeated over and over until the effective change would be 0 or minimal enough to have no real effect. Also please understand that the PR model we see where a PR6 page with 2 outbound (external) links to other pages passes PR3 through each link is completely invalid. Authority caps and link value scores are applied to give a natural curve in the PR model. Otherwise, no site could ever compete against a super authority site and the whole PR model would be forever skewed.

Be that as it may, only links that are complete, meaning links where the source and target URLs exist can have calculations applied. Dangling links and broken links (links to non-existing pages), are missing the requirement of a target page. It is impossible and impractical to pass link value to nothing.

So in your case, since the pages are restricted by the robots.txt file, there is no target URL and therefore no value passed.

What would be needed to further understand your scenario would be a better understanding of exactly what is being redirected and how. Anytime a series of redirects exist, the original page must be fetched to know this. There is a period of time as the search engines fetch these pages, measure the value of these pages, change the metrics, and compare targets against the robots.txt files, etc. All of this takes time and has the potential to be disruptive. It is possible that you are working through the process now.

My advice to people is to make life as simple for the search engines as possible and not to confuse things too much. Search engines like 404 errors and 301 redirects, however, a 301 redirect can have funny results sometimes and not as webmasters always expect. If it is possible to think this through a bit and simplify things, this may be a good opportunity to do so.

| improve this answer | |
  • Thanks for an extensive answer. However, what would you say about pages blocked in robots.txt and having solid backlinks to them, ranking very well on SERPs for good keywords? Google ranks them despite them being blocked and PR to them don't coming. How's that? They must be getting some link/authority juices flowing to them anyways, as they rank, and if they rank great, they must be impacting domain authority that way. No? – CamSpy Jun 29 '16 at 9:01
  • Dangling links can still appear in the SERPs. I argue that this is just plain flat-out wrong, however, Google has another opinion. Sometimes Google gets it completely wrong. This is one case. These can be known by A description for this result is not available because of this site's robots.txt. If you see a SERP link and snippet, it is not a dangling link - at least not yet. – closetnoc Jun 29 '16 at 23:55
1

If you stop the wrong URL´s from gettings accessed on your page, you might lower traffic or render backinglinks ineffective, yes

Example:If you block traffic to your e.g. contact page(/contacts.html), google and other search engines will get to know this and stop linking your contact page

You should only block URL´s on your robots.txt which´s content you dont want to have in the net.

Check if you have done this correctly and not accidentally included other important URL´s.

The redirection to other domains is rendering all backinglings to this content unvalid for the ranking of your page, cause bots dont see the relation of the backing link to your page any more and therefore rank the existing backlink for your other pages(your new domain). Therefore maybe cutting the link for your page(old URL) to other high value pages who were linking you. Without these backinglinks your page might have fall under some threshold were your contant generally is getting ranked lower because of not so much high value backing links.

Your options now could be to eather try reverting the changes. Or just work on building up new backing links.

| improve this answer | |
  • Please take a look at the comment I've posted under closetnoc's answer. – CamSpy Jun 29 '16 at 9:02
  • Pleas check developers.google.com/webmasters/control-crawl-index/docs/… (sorry dont know how to change language on this link) robots.txt , disallow -> no info of page content but still in ranking but decreases ranking. Use noindex if you want to block its appearing into search results. which do you use? – veritaS Jun 29 '16 at 9:58
  • I don't want to noindex these pages, indeed, I've blocked them in robots.txt because of them being Thin content or duplicates. However, Google thinks some of these pages are good enough so they get ranked and get traffic. Now, with all the info from me, can you take a look at my initial question? See what collision I have? Do you think that backlinks to robots.txt blocked pages boost domain's ranking, if these blocked pages rank? – CamSpy Jun 29 '16 at 10:21
  • you blocking a site in robots.txt will (block robtos to crawl it) therefore decreas the ranking slightly. If you now add backinglinks for this page, the ranking will increase for this sub-page as well as for your whole website. The block only prevents robots to scan content and use this content for ranking or keywords therefore does not call duplicate content. – veritaS Jun 29 '16 at 10:33
0

It makes sense to split your question up in two parts, each addressed separately.

Can backlinks to robots.txt blocked pages boost domain's ranking?

No, these backlinks shouldn't be affecting your domain's authority and thus rankings if they are pointed to pages which are blocked using robots.txt.

However, I can imagine search engines not completely discounting them because after all the links are a vote of confidence for your website.

Thus hurting our domain's ranking possibilities and SE traffic to our domain after redirecting such pages?

So in theory it shouldn't matter (that much) if you redirect these URLs to another domain. Unless my theory about search engines not completely discounting these links above is true, then it can.

To get a better understanding of the situation: what do you mean with 'significantly'? And was there any relation between the URLs that dropped in rankings (which currently still exists) and the URLs that were redirected?

Was anything else changed during the same period? In situations like this is very important to be able to rule out as much other reasons as we can.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.