Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms.

all the page urls like http://mydomain.com/it/page1, http://mydomain.com/en/page1 (for /it/ and /en/ just edit in /wp-admin/post.php?post=1&action=edit&message=1, create a Parent page - it or en, then save page under theme)

I stored photos in /thumb/ and /image/ folder. here is my full robots.txt.

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /thumb
Allow: /image

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

Sitemap: http://mydomain.com/sitemap.xml

User-agent: *
Disallow: /wp-login.php
Disallow: /wp-admin/
Disallow: /signup/
Disallow: /cgi-bin/
Disallow: /wp-
Disallow: /wp-admin/
Disallow: /signup/
Disallow: /cgi-bin/
Allow: /it/
Allow: /en/
Allow: /es/
Allow: /fr/
Allow: /de/

BTW, if i submit my site to https://www.google.com/webmasters/tools/robots-analysis-ac how many hours, google will crawl my robots.txt and sitemap.xml? and if I will apply for google adsense, must I first finish a seo? (google finish catch my robots.txt and sitemap.xml?) Many thanks.

share|improve this question
1  
There isn't really any need to disable all those directories, as they don't actually contain indexable content. What led you to try this? –  toomanyairmiles May 25 '12 at 18:23
    
@toomanyairmiles, i have read some articles about wp robots.txt, some author Disallow these folders, I am not sure yes or no, so ask a question. thanks. –  Giberno May 25 '12 at 18:33
    
Some do but there really isn't any point, none of the directories you've specified contain indexable content. I think some bloggers think this is a security feature, but it isn't. –  toomanyairmiles May 25 '12 at 18:51
    
@toomanyairmiles, i see you have a higher reputation score, so in your opinion, what should i do the robots.txt is best? and if i submit my site to google.com/webmasters/tools/robots-analysis-ac how many hours, google will crawl my robots.txt and sitemap.xml? and if I will apply for google adsense, must I first finish a seo? (google finish catch my robots.txt and sitemap.xml?) –  Giberno May 25 '12 at 19:26
    
You don't need to do anything with it. The above has no real benifit for your site, leave the sitemap address in but that's all. Google will get round to your site, but there are no guarantees on timescale I'm afraid. –  toomanyairmiles May 25 '12 at 20:48
add comment

1 Answer

up vote 1 down vote accepted

I would be cautious only allowing Google image bot to your image folders. We don't know if Google's user agent will simply be Google or how their search bots contribute to their image search results.

The latest version of WordPress also includes I believe index files for folders in case you have world readable folders.

How do you plan to handle your wp-content folders? I would suggest you simply let Google crawl your entire site, include your sitemap file in the robots if anything. And just run a site: search occasionally to see what has been indexed. But a properly configured server and htaccess file should prevent unwanted files from getting indexed.

Do you run any CGI scripts? If not just delete that folder you don't need it.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.