Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

A friend of mine has his site at domain.com. He, because of a framework and blah blah blah, has allocated the content inside a folder named /home. So, if you go to domain.com it does a redirect to domain.com/home. It is a permanent redirect.

Now, when I go to Google, Bing, Yahoo Search, etc, and look for "site:domain.com" no pages are listed but the domain.com page and there is no page there actually (the page is inside /home).

Can the permanent redirect be the reason why no pages inside /home are indexed?

If you need to look inside the site let me know so I let the real domain here. Thank you!

EDIT:

I've went through the index status report at Google WebMaster tools and I get this report by today. Does it say anything interesting?

Ever crawled: 2,874 Total indexed: 2 Blocked by robots: 1 Removed: 0

share|improve this question
1  
Is your site indexed at all? It should still return results under /home even with site:domain.com. Check your robots.txt maybe someone forgot to remove Disallow: * –  edvinas.me Mar 26 at 19:39
    
@edvinas.me thanks for taking the time to look at this. The site robots is at reclutec.com/robots.txt - It is allowing everything and disallowing two folders content. Do you notice anything wrong? –  Marco F. Mar 26 at 19:47
    
IF you have access to the Apache configuration files, perhaps it would be better to change the directory of your site to include the home directory that way a redirect would not be necessary. –  closetnoc Mar 26 at 19:55
1  
How long has the site been up and active for? –  nathangiesbrecht Mar 26 at 20:16
    
Was the site ever active at the root, and not inside /home? Because I've noticed that the logo links to an almost identical page at reclutec.com/home/main/display. If the homepage was originally in the root, it could be that /home/ and /home/main/display are both considered "duplicate content" and Google's sticking with it's original URL for that content. Though I'm not so sure about that since you do seem to have the Canonical tags implemented correctly. –  nathangiesbrecht Mar 26 at 20:36
show 3 more comments

3 Answers 3

The robots.txt is OK, there is no need for the Allow part, but that is still OK.

Although the domain has been registered some time ago, the content doesn't seem to be very old, or the site has almost no activity, so I'll assume that is being built or rebuilt recently.

The lack of content and the time when you sent the sitemap to the crawlers affect the indexing process, so most probably you have to wait a bit longer.

About the question itself, yes, redirects affect SEO, but not noticeably, there are many messages everywhere about the leak of juice after a 301, but as I mention and you can read, it's not relevant, even less inside your own domain/site.

Also, consider that on Goggle you have few pages, but on Bing and Yahoo there are more results, most of the results are pages with more content than the homepage and the few I checked manually, which helps you to see that search engines are more interested on content than number of pages.

Update after you question update Check the article, Index status on Google Webmaster Tools, it may help you see the difference between crawled and indexed. Or this other one, Google “Reveals Index Secrets”: Charts Indexing of Your Site Over Time on Search Engine Land.

share|improve this answer
    
Thanks @PatomaS. The sites has almost two years live. - I have noticed that the description of the home page in the search results is not accurate to the meta description set in the live site. Does it say anything? I'm very confused about this, what could be the real reason if it isn't the permanent redirection? –  Marco F. Mar 27 at 20:32
    
If you are talking about the text that crawlers use on the search result page, they use fragments of text on the page that are relevant to the keywords used for the query, if there are no words, they use the description. If search uing the site: filter, they may use any of the above. –  PatomaS Mar 28 at 0:46
add comment

By what I can tell, by the comments and the link to the site, the robots.txt is correct for the site to index the /home/ folder (I could be wrong).

Can the permanent redirect be the reason why no pages inside /home are indexed?

The 301, permanent redirect, tells the Search Engine crawlers where to go, which will be to the /home/ folder to index the site.

If you have access to edit the robots.txt file, I would suggest to change it to this to help you out some more.

User-Agent: *
Disallow: /home/spa/*
Disallow: /home/eng/*

From bushing up on Robots.txt here is how it works:

  1. User-Agent: * says that it is going to allow all search engines to index the website.
  2. We are going to keep them from indexing the two folders which you did not want to index.

EDIT 1

By looking into the Allow which was in the robots.txt, there is no case to use Allow or is it apart of a Robots.txt file. I believe that was causing an error of your site getting index with the search engines.

share|improve this answer
    
Are you saying that Allow is not allowed in robots.txt files? Because it most certainly is. Look at stackexchange.com's robots.txt or Google's to see evidence of it in action. –  nathangiesbrecht Mar 26 at 20:16
    
I know some people do use it, but I don't see to many cases where it is actually needed myself. Unless I am missing something about it completely. –  user37204 Mar 26 at 20:19
    
In this case it's not needed, but I don't think that's what's causing the issue. The use-case for it is if you wanted to do something like Disallow: /images/ Allow: /images/thumbnails/ to disallow bots from crawling /images but allow them to crawl /images/thumbnails. It's also useful if you want to block all bots from /images but allow GoogleBot-Images to crawl /images by specifying different User-Agent blocks in the robots.txt –  nathangiesbrecht Mar 26 at 20:21
    
Thanks @Traven for checking this out. I want all pages indexed expect for the ones with the pattern /home/{lang}/*. From what you say, this is not the rule since Allow: * is overriding and taking priority to the next two Disallow: ... lines. So, I will request a modification on this file. Although, this is not answering my question since the robots file is not actually preventing the pages to be indexed, and what I want is to have all the pages indexed. Make sense? –  Marco F. Mar 27 at 20:14
add comment

Normally, I would say no. I am assuming that the redirect is interrupting access to the site somehow that you may not be aware of. I do not have an example of the redirect that you are using so there is a huge information vacuum in your question in that regard.

I reread everything here and I noticed that there was no redirect to answer your question. So I got busy.

Apache configuration files are either under /etc/apache2/ or /etc/local/apache2/. It is likely the files you need to edit are under /sites-available. If there is only one site on the server, the default file is the one you want to edit. Otherwise, if there are several sites, the file you want to edit will have a name format of domainname.tld.conf.

In this file, there will be a Directory directive that will point to the directory of the site. Edit this and add /home/.

If the Apache configuration files are not available to you, Here is an example redirect that may work. I have not tested it.

RewriteCond %{REQUEST_URI} !^/home/.*$ [NC]
RewriteRule ^(.*)$ http://www.example.com/home/$1 [L,R=301]

But I suspect it should work okay. You will notice that I put a condition in so that the redirect would only work where the /home directory does not exist.

Let us know how it works. I can edit the answer to update it with new ideas as we go or someone can offer a different redirect.

share|improve this answer
    
His redirect works. That's not the problem. The problem is the way that search engines are indexing the redirected homepage. –  nathangiesbrecht Mar 27 at 21:21
    
I did miss that point, but I was assuming that the redirect was not search engine friendly somehow. Can you re-edit the question and give us an example of the redirect? This may help us to know what is going on better and finally give you some pointers. –  closetnoc Mar 27 at 21:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.