• I added my url to Google http://www.google.com/addurl.html

  • I added a meta tag with description and keywords

  • I added it in Google web master tools and submit the site map

  • There is no any robots in my site

But when I create the sitemap.xml automatically with http://www.xml-sitemaps.com/ or other tools they don't find all the pages!

Are there any better tools, how can I fix the problem?

  • 1
    [This is two questions, by the way; try to keep threads to a single topic.] How long have you waited after creating/submitting your site? It's not automatic. For the second part about your sitemap, see this previous question.
    – Su'
    Commented Dec 24, 2011 at 15:42
  • Are you using a CMS like wordpress? Commented Jan 25, 2012 at 10:30
  • And remember, meta information is either completely antiquated, or moderately antiquated. As John indicated, quality, relevant backlinks from quality, relevant directories and websites is the #1 rule in SEO. See my post here for more information webmasters.stackexchange.com/questions/23810/… Commented Nov 12, 2012 at 19:16
  • For WordPress, Slim SEO is a great lightweight option. Will generate for you + do some other nice things. Commented Oct 26, 2021 at 3:16

7 Answers 7


You're doing the right things. Keep up the good work, and don't get too antsy. If you did it right, the googlebot will come. In the meantime, consider this free product: http://www.vigos.com/products/gsitemap/ ... I'm not an advertiser for 'em. I just use the product.

It generates a perfect sitemap. You'll have to make some manual adjustments once it's completed, and delete things you don't want spidered. Also, understand the Google Sandbox. Here's an SEO article I wrote awhile back:

The Problem: I hear many people ask why they're not listed in Google yet, and I wanted to offer a posting that clears the confusion of why this is happening. There is no doubt anymore about the existence of The Google Sandbox, which came into existence in March of 2004 in an effort to clamp down on SPAM websites, among other things.

The Symptoms: Basically, if your website is relatively new, and you're ranking in Yahoo, MSN, and others, but not Google, you're caught in the "Google Sandbox", which is referred to in other terms such as "Google Time Delay", or "Google Age Delay". Most refer to it as the "Google Sandbox", however.

What is the "Google Sandbox"? This is a probationary period where your website is checked out. The Googlebot will index your website, but will not list you immediately. Do not panic! Things will be okay, although there is really nothing you can do to get out of the Sandbox or avoid it. Some people will suggest putting your site on a subdomain of an old trusted domain, and then setting up a 301 redirect to your new website after google has indexed the subdomain.


There must be links to the pages somewhere in your site. The sitemap generator will start at the root and follow all links on the pages and so on until it has exhausted all links. If there are no links to the page it will not be found.

You could manually add the entries to the XML file to create a complete sitemap.

Alternatively create a static page in your site with the links to enable the sitemap generator to find them if there is no genuine reason to link to them.


but in creating the sitemap automatically with that site http://www.xml-sitemaps.com/ and others it dosn't reach all pages!

I'm assuming the question is, "Why aren't all of the pages of my website indexed by the search engines (or specifically Google)?

Just because you submit your website to the search engines, or provide an XML sitemap, doesn't mean all of your pages are guaranteed to be crawled and/or indexed or included in their search results. By notifying the search engines of your website and all of your available pages you give them the opportunity to find and crawl your content. But they have the final say about whether to include and how to rank your content. The best way to influence this is to get quality links to point to your pages (both your home page and internal pages). The higher the quality of the links, the more pages you'll get indexed and the better they'll rank. (And to take it a step further, if you do a good job with your internal links your well ranked pages can help prop up other pages in your website).

  • Google does find some pages on a website to have no relevance and refuses to index them. As John points out, rethink the link structure to and content of those pages. One thing that can cause this is having too deep a link path, these pages are viewed as internal supplemental information to their parent page and if that's the only link, may not be seen to be relevant search fodder. Commented Dec 24, 2011 at 18:59
  • Good tips, but it sounds more like he's saying the sitemapping tools don't reach all pages. This is indicative of broken internal link structure. There are workarounds with other sitemap generators (such as a side project by some Google developers that monitors user traffic instead of crawling the site), but it's still better to fix the internal link structure so all pages are crawlable, with higher priority pages taking less clicks to reach than less important ones. Commented Jan 25, 2012 at 13:35

Try Xenu's Link Sleuth, it's a much better tool than xml site maps, and can generate sitemap style link lists.


txt file via Google webmaster account and put it in root domain.It will enable google crawlers to spider your contents and If you use CMS you can create XML sitemap with some plugins available.


It's 2012 you don't have to submit your website to search engines. Google is a registrar and knows when new websites are registered. If they decide to crawl you is another matter.

Google doesn't pay any attention to meta keywords and often changes the meta description they display in search results based on the search executed.

Verifying your website and submitting sitemaps is good. Most free sitemap generation tools have limits.

If you are not using a CMS with a built in sitemap generation tool then $19.99 for the unlimited version of http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html is well worth it. It will crawl all your pages.

If you have submitted a sitemap to Google, and you are not blocking robots. Then there is no problem. Simply put you have to be patient and wait for Google to crawl, index, and cache your websites pages.

  • This is still a great answer if the site has < ~10-20k pages. Though for anyone curious enough to register Search Console, I can't imagine not just submitting it. Commented Oct 26, 2021 at 3:12

If you're creating the sitemap and manually uploading to your server, you might want to check the permissions on files in your site's root directory.

If these are off, it is possible that googlebot (or others), crawlers, or a site like xml-sitemaps.com can't access the file.

I personally recommend 640 for files (many use 644)

find html/ -type f -exec chmod 640 {} \;

For folders I use 750 (many use 755)

find html/ -type d -exec chmod 750 {} \;

If copying, replace html/ with your site's folder.