Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google Webmaster Tools for more than 2 months to little avail. I want to know if I have some blatantly wrong configuration somewhere that is causing search engines to be unable to index this site. I noticed a similar issue with another website I searched for online (ECEGSA - The University of British Columbia at gsa.ece.ubc.ca), making me believe this might be a concern that people might be looking an answer for.

Here are the details: The website in question is: http://gsa.ece.umd.edu/. It runs using Joomla 2.5.x (latest). The site was up since around mid December of 2013, and I noticed right from the get go that the site was not being indexed correctly on Google. Specifically I see the following message when I search for the website on Google:

A description for this result is not available because of this site's robots.txt – learn more.

The thing is in December till around March I used the default Joomla robots.txt file which is:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

Nothing there should stop Google from searching my website. And even more confusingly, when I go to Google Webmaster tools, under "Blocked URLs" tab, when I try many of the links on the site, they are all shown up as "Allowed". I then tried adding a sitemap, putting it in the robots.txt file. That did not help. Same exact search result, same behavior in the "Blocked URLs" tab on the webmaster tools. Now additionally, the "sitemaps" tab says for several links an error saying "URL is robotted out". I tried those exact links in the "Blocked URLs" and they are allowed!

I then tried deleting the robots.txt file. No use. Same exact problem.

Here is an example screenshot from Google's Webmaster Tools: Index Status for my website showing no crawl errors and no blocked URLs, in direct contradiction to what the sitemap tab says and what the search result says! FRUSTRATION!

At this point I cannot give a rational explanation to why this is happening and neither can anyone in the IT department here. No one on Joomla forums can seem to understand what is going on.

Based on what I explained, does it seem that I have somehow set a setting in the robots.txt or in .htaccess or somewhere else, incorrectly?

share|improve this question
    
You have more than 20k links on the site that have 3xx response: 301, 302, 303. I think this is the possible reason why G doesn't index your site. –  Oleg May 25 at 6:27
    
    
Is there a way to avoid a 3xx redirect while still using the CAS login plugin? What other ways can I use to redirect people to the login page managed by my university? –  Abhijit May 25 at 15:18
    
@unor sorry for the cross post. However, I was not certain if the issue was pertinent to Joomla OR if the issue was a site configuration error that was more webmaster centric. So I posted it at both places. –  Abhijit May 25 at 15:37

2 Answers 2

up vote 3 down vote accepted

You are 303 redirecting to a login page which then redirects back:

$ curl --head http://gsa.ece.umd.edu/
HTTP/1.1 303 See other
Date: Sun, 25 May 2014 10:45:59 GMT
Server: Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4
X-Powered-By: PHP/5.4.21
Set-Cookie: c32b08af5ad5c16062381828f6a1b64e=6d66a9870a7067c444930d8deb190cd9; path=/
Location: https://login.umd.edu/cas/login?service=http%3A%2F%2Fgsa.ece.umd.edu%2F%3Fserver%3D1&gateway=true
Content-Type: text/html; charset=utf-8

The login page to which you are redirecting is blocked by robots.txt:

$ curl -s http://login.umd.edu/robots.txt
User-agent: *
Disallow: /cas/
share|improve this answer
    
Oh! I obviously don't have control to the login page. And I need to use the CAS login page as it is the best way to login to my website, as I want students to use their university login. I used a plugin for Joomla to do the CAS login. Is there a way to avoid a 303 redirect? By the way, thank you SOO much for this info. I finally feel relieved to know why this might be happening! :) –  Abhijit May 25 at 15:15
    
I'm not sure why a login plugin would be redirecting before the user clicks the login link. There may be some configuration that you can tweak, or a different plugin with similar functionality that you can use. –  Stephen Ostermiller May 25 at 22:12
    
Ok, I will have to look into it. I will ask the IT staff here if they have a nicer solution to the login. It is quite a pain to work with CAS as it is not so clearly documented. In the meantime I will probably remove the login portion to see if it helps remove this issue. –  Abhijit May 25 at 23:30
    
Thanks a lot Stephen. Indeed removing the offending plugin has solved the problem and Google has finally placed a blurb underneath my site result. Now, I have an added problem of authentication that plays well with SEO. Oh well, baby steps! –  Abhijit May 26 at 0:36

Thank you Stephen, for your help. Your observation was correct and indeed it was due to the 303 Redirect that the search engine indexing was not working correctly.

I want to add to this discussion and point people to my answer on, Joomla Stackexchange: Bad configuration of Externallogin extension causing search engine indexing problem, which clearly explains what the configuration error is with regard to the plugin externallogin that will cause this issue, so that others who may encounter this issue know how to fix it.

FIX
In short, to fix this issue, if you ever encounter it, you must correct the configuration of the plugin as follows:
1. In the extensions menu, under "External Login>", go to the server configuration.
2. In the connections tab, for "Automatic Login/Logout", choose 'No'.

The error occurs if you choose 'Yes' in step 2, since in this case, the extension automatically adds a 303 redirect to all pages of your website, to check whether a user has been logged in to the CAS in the browser session, and if so it automatically logs you into the site. This feature will cause search engine indexing problems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.