1

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse.

For some stupid reason Google can't crawl my pages and gives me warnings, that my urls in my sitemap.xml are blocked by the robots.txt, even if I allow Google to crawl my whole site with Disallow:. My .htacces rules working fine on my homepage.

robots.txt

User-agent: *
# allow sitemap
Allow: /sitemap.xml

# allow article page
Allow: /blog/article/

# allow article 1
Allow: /blog/article/first-article

# sitemap url
Sitemap: http://www.my-page.de/sitemap.xml

sitemap.xml

...
<url>
  <loc>http://www.my-page.de</loc>
  <changefreq>daily</changefreq>
</url>
<url>
  <loc>http://www.my-page.de/blog/article</loc>
  <changefreq>daily</changefreq>
</url>
<url>
  <loc>http://www.my-page.de/blog/article/first-article</loc>
  <changefreq>daily</changefreq>
</url>
...

.htaccess

# ERROR DOCUMENTS
ErrorDocument 401 /includes/errors/404.php
ErrorDocument 403 /includes/errors/404.php
ErrorDocument 404 /includes/errors/404.php
ErrorDocument 410 /includes/errors/404.php

# ACTIVATING REWRITEENGINE
RewriteEngine on

# TRANSLATE NON-WWW TO WWW
RewriteCond %{HTTP_HOST} !^www\.
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]

# TRANSLATE EVERY SUB_DOMAIN_PAGE TO DOMAIN_PAGE
RewriteCond %{HTTP_HOST} ^www.mypage.de$   [NC]
RewriteRule ^ http://www.my-page.de  [L,R]

# CLEAN URLS
# blog & article
RewriteRule ^(.*)blog/article.?$ $article.php
RewriteRule ^(.*)first-article$ first-article.php

root directory

├─── ...
├─── blog
│    └─── first-article.php
├─── sitemap.xml
├─── robots.txt
├─── .htaccess
├─── article.php
├─── ...

EDIT: For some reason Google now managed to crawl through the URLs, listed in my sitemap. I changed absolutely nothing. Maybe the errors were caused due to the fact @closetnoc mentioned below (Google checks the robots.txt file periodically).

  • Sitemap: my-page.de/sitemap.xml is wrong. Remove it. Also, not sure what Allow: /$ is supposed to do for you. Most people get into trouble by disallowing all first. It is usually far simpler and more successful to disallow specific sections only. BTW- your sitemap has nothing to do with this problem. It may be better to include your .htaccess to make sure there is not something wrong there. – closetnoc Mar 23 '16 at 22:12
  • Memorized. But as mentioned, even if I allow the crawlers to crawl every part of my page I get the same warnings/errors. The robots.txt shouldn't be the problem then, right? – CodeVolunteer Mar 23 '16 at 22:17
  • You can test this quickly by blanking out your robots.txt file. – closetnoc Mar 23 '16 at 22:22
  • I deleted everything from the robots.txt and I get the same warnings again. So the problem is either my .htaccess or directorie I guess, is it maybe caused by the fact that a part of my re-written url's are also a directory in my root-folder? – CodeVolunteer Mar 23 '16 at 22:33
  • 1
    You should use Google Search Console to test this. Google only checks the robots.txt file periodically, once per 24 hours generally, and otherwise may not be re-reading the file. Use the Search Console to test your robots.txt file and use Fetch as Google to ensure it can read your pages without using any cached robots.txt file. You may want to test a sample set of pages to ensure that it all works. BTW- your last entry in your robots.txt file is clearly not right. Whole URLs are not understood. – closetnoc Mar 23 '16 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.