1

A friend of mine has a site where he allocated the content inside a folder named /home. He applied a redirection rule so that if you go to domain.com it does immediately go to domain.com/home. It is a permanent redirect since there isn't any content at the root folder.

Now, when I go to Google, Bing and Yahoo Search, and look for "site:domain.com" no pages are listed. The only page listed is the one with a link to the root folder at domain.com, and, as I mentioned above, there is no content or physical page at root, all the content is at the /home folder.

Can the permanent redirect be the reason why no pages inside /home are indexed?

EDIT:

I've gone through the index status report at Google WebMaster tools and I get this report by today. Does it say anything interesting?

Ever crawled: 2,874, Total indexed: 2, Blocked by robots: 1, Removed: 0

ABOUT SELECTED ANSWER:

As @PatomaS explains, the redirection was not be the reason why the pages where have not been indexed. It was just about time. Although, I am not sure if the redirection caused any delays on the indexation process. My friend had to wait more than 3 months to see his site in the search results, what I think was a long time.

8
  • 1
    Is your site indexed at all? It should still return results under /home even with site:domain.com. Check your robots.txt maybe someone forgot to remove Disallow: *
    – ek9
    Commented Mar 26, 2014 at 19:39
  • @edvinas.me thanks for taking the time to look at this. The site robots is at reclutec.com/robots.txt - It is allowing everything and disallowing two folders content. Do you notice anything wrong? Commented Mar 26, 2014 at 19:47
  • IF you have access to the Apache configuration files, perhaps it would be better to change the directory of your site to include the home directory that way a redirect would not be necessary.
    – closetnoc
    Commented Mar 26, 2014 at 19:55
  • 2
    How long has the site been up and active for? Commented Mar 26, 2014 at 20:16
  • Was the site ever active at the root, and not inside /home? Because I've noticed that the logo links to an almost identical page at reclutec.com/home/main/display. If the homepage was originally in the root, it could be that /home/ and /home/main/display are both considered "duplicate content" and Google's sticking with it's original URL for that content. Though I'm not so sure about that since you do seem to have the Canonical tags implemented correctly. Commented Mar 26, 2014 at 20:36

3 Answers 3

1

The robots.txt is OK, there is no need for the Allow part, but that is still OK.

Although the domain has been registered some time ago, the content doesn't seem to be very old, or the site has almost no activity, so I'll assume that is being built or rebuilt recently.

The lack of content and the time when you sent the sitemap to the crawlers affect the indexing process, so most probably you have to wait a bit longer.

About the question itself, yes, redirects affect SEO, but not noticeably, there are many messages everywhere about the leak of juice after a 301, but as I mention and you can read, it's not relevant, even less inside your own domain/site.

Also, consider that on Goggle you have few pages, but on Bing and Yahoo there are more results, most of the results are pages with more content than the homepage and the few I checked manually, which helps you to see that search engines are more interested on content than number of pages.

Update after you question update Check the article, Index status on Google Webmaster Tools, it may help you see the difference between crawled and indexed. Or this other one, Google “Reveals Index Secrets”: Charts Indexing of Your Site Over Time on Search Engine Land.

2
  • Thanks @PatomaS. The sites has almost two years live. - I have noticed that the description of the home page in the search results is not accurate to the meta description set in the live site. Does it say anything? I'm very confused about this, what could be the real reason if it isn't the permanent redirection? Commented Mar 27, 2014 at 20:32
  • If you are talking about the text that crawlers use on the search result page, they use fragments of text on the page that are relevant to the keywords used for the query, if there are no words, they use the description. If search uing the site: filter, they may use any of the above.
    – PatomaS
    Commented Mar 28, 2014 at 0:46
1

By what I can tell, by the comments and the link to the site, the robots.txt is correct for the site to index the /home/ folder (I could be wrong).

Can the permanent redirect be the reason why no pages inside /home are indexed?

The 301, permanent redirect, tells the Search Engine crawlers where to go, which will be to the /home/ folder to index the site.

If you have access to edit the robots.txt file, I would suggest to change it to this to help you out some more.

User-Agent: *
Disallow: /home/spa/*
Disallow: /home/eng/*

From bushing up on Robots.txt here is how it works:

  1. User-Agent: * says that it is going to allow all search engines to index the website.
  2. We are going to keep them from indexing the two folders which you did not want to index.

EDIT 1

By looking into the Allow which was in the robots.txt, there is no case to use Allow or is it apart of a Robots.txt file. I believe that was causing an error of your site getting index with the search engines.

4
  • Are you saying that Allow is not allowed in robots.txt files? Because it most certainly is. Look at stackexchange.com's robots.txt or Google's to see evidence of it in action. Commented Mar 26, 2014 at 20:16
  • I know some people do use it, but I don't see to many cases where it is actually needed myself. Unless I am missing something about it completely.
    – user37204
    Commented Mar 26, 2014 at 20:19
  • In this case it's not needed, but I don't think that's what's causing the issue. The use-case for it is if you wanted to do something like Disallow: /images/ Allow: /images/thumbnails/ to disallow bots from crawling /images but allow them to crawl /images/thumbnails. It's also useful if you want to block all bots from /images but allow GoogleBot-Images to crawl /images by specifying different User-Agent blocks in the robots.txt Commented Mar 26, 2014 at 20:21
  • Thanks @Traven for checking this out. I want all pages indexed expect for the ones with the pattern /home/{lang}/*. From what you say, this is not the rule since Allow: * is overriding and taking priority to the next two Disallow: ... lines. So, I will request a modification on this file. Although, this is not answering my question since the robots file is not actually preventing the pages to be indexed, and what I want is to have all the pages indexed. Make sense? Commented Mar 27, 2014 at 20:14
-1

Normally, I would say no. I am assuming that the redirect is interrupting access to the site somehow that you may not be aware of. I do not have an example of the redirect that you are using so there is a huge information vacuum in your question in that regard.

I reread everything here and I noticed that there was no redirect to answer your question. So I got busy.

Apache configuration files are either under /etc/apache2/ or /etc/local/apache2/. It is likely the files you need to edit are under /sites-available. If there is only one site on the server, the default file is the one you want to edit. Otherwise, if there are several sites, the file you want to edit will have a name format of domainname.tld.conf.

In this file, there will be a Directory directive that will point to the directory of the site. Edit this and add /home/.

If the Apache configuration files are not available to you, Here is an example redirect that may work. I have not tested it.

RewriteCond %{REQUEST_URI} !^/home/.*$ [NC]
RewriteRule ^(.*)$ http://www.example.com/home/$1 [L,R=301]

But I suspect it should work okay. You will notice that I put a condition in so that the redirect would only work where the /home directory does not exist.

Let us know how it works. I can edit the answer to update it with new ideas as we go or someone can offer a different redirect.

2
  • His redirect works. That's not the problem. The problem is the way that search engines are indexing the redirected homepage. Commented Mar 27, 2014 at 21:21
  • I did miss that point, but I was assuming that the redirect was not search engine friendly somehow. Can you re-edit the question and give us an example of the redirect? This may help us to know what is going on better and finally give you some pointers.
    – closetnoc
    Commented Mar 27, 2014 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.