Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a small website with about 20 pages with .html extensions, containing unique articles I wrote consisting of about 1000 words or more. These 20 pages are the main feature of my site.

Root Directory

My root directory is structured in such a way that all 20 pages of my HTML files are in the root directory, not contained in any folder. I also have a forum installed on my website, and a PHP/jQuery poll plugin which is kept in the root directory in their two respective folders and this is where the problem comes in

My Problem - Question 1

I mentioned earlier I only have 20 HTML files which makes up my site. However, when I do a scan of my site to generate a site map more than 70 pages shows up, yet only 20 of those 70+ pages are my own original pages I want crawled and indexed. The other 50+ files contained in the PHP poll folders and PHP forum folders are all source code files and not unique and do not contain any real content.

Should I or can I block Google from scanning those two folders containing the poll and forum files? If yes, how would I do this in my robots.txt file?

Question 2

If I have these poll and forum folders in my root directory, and a scan of my site shows 70+ files but only 20 pages of those are my own would it have any effect on my SEO?

I have purchased GoDaddy SEO visibility service with my hosting, a crappy but basic product service. When I started my site and only had those 20 HTML files, I got a SEO score of 85+ after installing my forum and poll and re-evaluating my sites SEO score the pages increased to 70+ and my SEO score dropped from 85+ to the mid 50s. I realize these tools are not always accurate but it did prickle my curiosity, leading me to ask this question.

I would also like to add that my forum is rather empty at this stage, but I do enjoy the poll plugin.

Conclusion

So what should I do:

  • Block robots.txt from scanning those folders if so how?

  • Should I just remove the forum and poll folder?

  • Should I not worry about it, and accept products giving you a SEO score is crap?

share|improve this question

migrated from stackoverflow.com Feb 18 at 16:02

This question came from our site for professional and enthusiast programmers.

    
What is a "SEO score"? –  unor Feb 19 at 21:37

1 Answer 1

up vote 1 down vote accepted

To prevent robots indexing folders or files, create a robots file (called Robots.txt) with the following content and upload it to your root directory

User-agent: *
Disallow: /myFolder1/
Disallow: /myFolder2/
Disallow: /myFolder3/

Or, you can do it by files as well.

User-agent: *
Disallow: /MyFolder/other.html
Disallow: /that.html
Disallow: /MyFOlder/OtherFolder/this.html

Please note, the robots file only works if the robot listens to this file!

In regards to SEO performance, that is very hard to answer. The SEO tools should be used with a pinch of salt - no one knows how to get to position 1, so I'm not sure what the tools are promising they can do. However, use them alongside analytical tools and see how it is affected.

Since your site is new, personally, I wouldn't block anything and see how it does over the next few months. If within 6 months, then consider altering the content.

I've never had to block certain pages (other than error message pages or login pages etc) and I don't think doing so is the answer. Personally, I'd leave it exposed to the engines and let them decide...

share|improve this answer
    
Thank you for taking the time to answer if I already have a robots.txt file shall I just add those folder names to the existing content allready in the file? IF I decide to go that route. –  Marilee Feb 18 at 13:52
    
@Marilee Yes, you need to update the robots file as you see fit, based upon your needs really. You only ever have 1 robots.txt file –  Dave Feb 18 at 13:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.