93

A Sitemap file helps search engines to discover new and updated URLs on your website. In particular, if your website is fairly large, then this can help them to be able to focus on the new & updated content, instead of having to blindly crawl through everything to see if anything has changed. That can result in new content being found much faster, which ...


18

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. The recommended method to ...


12

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


11

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature. You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...


10

The value of the xmlns attribute is to uniquely identify a namespace (the "namespace URI"). It does not relate to whether your site is serving content over HTTP or HTTPS - for this it makes no difference. The sitemaps.org site defines the protocol using http://www.sitemaps.org/schemas/sitemap/0.9 as the recognised namespace - so this should really be the ...


9

The first step would be to detect the User-Agent of the bots you want to allow, and serve a different file if it is not a User-Agent that you want to allow. For example, you could have two versions of robots.txt, one with and one without a reference to the sitemap, so your competitors won't find the sitemap if they look inside your robots.txt. Then, you ...


9

Sitemaps not sitemap when dealing with high volume of content Your first mistake would to assume that you use 'a sitemap', when dealing with sites that contain a lot of content you need to think in terms of 'sitemaps'. Google indexes content without a sitemap Now it's worth mentioning that Google will pretty much index any important URL without the need ...


9

MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?) <Files ~ "sitemap.*\.xml(\.gz)?$"> Header append X-Robots-Tag "noindex" </Files> Under nginx ...


8

The only place on your site where you might place a link (or URL reference) to your XML sitemap is in your robots.txt file. This will allow Google and all the other search engines you've not explicitly submitted a sitemap to, that support this extended robots.txt syntax, to find your XML sitemap: Sitemap: http://example.com/Sitemap.xml Absolute or relative ...


8

It is not possible to define any hierarchal structure in your XML sitemap. The XML sitemap is a straight forward list of all your pages. Any hierarchal structure to your pages will be determined by Google when it crawls your site. The hierarchal structure is more relevant to your users. So, your HTML sitemap (if you have one) could be defined in this way. ...


7

Google periodically checks your submitted sitemap.xml file for updates. You only need to submit it once. See: How often does GWT check dynamic sitemaps?


7

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag ...


7

As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap limit, not a website limit. (The file must also be no larger than 50MB*1 (uncompressed) - so whichever comes first.) (*1 Previously 10MB.) Then you can also have ...


6

It's depends on which search engine you refer to. Most people are interested in google so I'll assume that's what you meant: Generally speaking, the higher your rank is, and if the frequency in which you change your site (add articles etc) is high - your site will be crawled more frequently. The following article might interest you.


6

If you're not getting any errors then you can assume Google has parsed it and is aware of the contents. But that doesn't mean they will crawl and/or index those pages. Sitemaps are just another way to tell search engines about your pages. They are not obligated to crawl and index any or all of those pages. The same applies to them finding pages through links ...


6

Yes, sitemap.xml.gzis a valid name. The .gz extension just means that it's been compressed (using gzip compression), so that it's smaller and served faster. Most search engine bots can read gzip'd compressed content. You can edit the sitemap's XML content just by decompressing sitemap.xml.gz using a compression utility like 7-zip. "Robots.txt" is ...


6

No, neither is required. By default your site will be indexed by Google. The robots.txt file is useful for preventing it from accessing specific directories or files. It's not a security mechanism, however, and if you don't want the public to access those pages/files, you should block access another way, such as an .htaccess file. The sitemap.xml is also ...


6

You are asking two questions here. Does a sitemap need to be XML? The simple answer is no, it doesn't have to be XML. It can be XML file, a text file or RSS/Atom feed (which is basically XML), HTML Sitemap HTML Sitemaps: These are used on your website to display the layout in layers on your website to any customer that would wish too (don't know why they ...


6

The problem is that if you (quite rightly) want your content to be indexed by search engines, anyone who performs a site: search in one of the search engines will be able to see what URLs are indexed. If you want to "hide" your sitemap you could have it on a URL with a "secret" name so it's not obvious to anyone who may be looking for it, but seeing as it'...


6

Yes. Everything you ask for is possible. And here's an example XML sitemap file generated by the Drupal XML sitemap module with a little bit of configuration. Everything is done for you out of the box. http://softkube.com/sitemap.xml If you check the code of the XML file there's a link to an XSL and inside that file you can see the code with references to ...


6

As others have stated, a sitemap isn't necessary for such a small amount of content. If you really want to have one, then including everything on one sitemap is okay. Can you combine an XML Sitemap with an Image XML Sitemap? You can use a separate sitemap to list images, or you can add image information to an existing sitemap. Can you combine an XML ...


5

Let's try an empirical approach. In the access logs for my site, I see 55 sitemap requests over the last 33 days. Out of those 55, 30 are from Googlebot, 21 from msnbot and the remaining four are from Exabot. (I've only submitted the sitemap manually to Google; the others have found it through robots.txt.) So that's one data point for about "every day", ...


5

In my experience: with no sitemap submitted, it might take more than 30 days, with sitemap it usually takes a couple of weeks.


5

There is no fundamental disadvantage to text-only sitemaps aside from what you've touched on. Setting the priority and last-modified date can affect the crawl frequency and on a large site with both frequently and infrequently changing content that can be very important. XML sitemaps can also carry media information (e.g. video sitemap) and if you have ...


5

Search engines will crawl your site more frequently if your content changes more often. If you're just adding new URL's, then there's no need to regenerate your sitemap.xml each time. However, you should update the last modification date in your sitemap by modifying the lastmod attribute, and specify the changefreq for any URL's that will periodically ...


5

You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines. If you have property/form URLs in your sitemap, you've probably used a 3rd party Sitemap Generator. These just crawl all links on your website indescriminately, whereas something like https://www....


5

Not long ago I was in the same boat when using WordPress plugin Yoast SEO it produced several sitemaps and I was curious to know which one to load to Google's Webmaster Tools Search Console I ran across this article on Google regarding Search Console Help: Simplify multiple sitemap management If you have many sitemaps, you can use a sitemaps index file as ...


5

You need to disable and reenable the sitemap, going to SEO > Sitemap XML. It probably will solve your problem (Yoast bug).


5

You can validate your sitemap on https://www.google.com/webmasters/tools. It will give an error when you use https to refer to the schema. I have also seen other sitemap validators that refuse to validate the https version of the namespace, so using http might be the only correct option, even when your site only uses https.


4

There is no timetable for when Google will crawl and/or index your pages. For new sites it usually is a not very quick process unless you happen to get a high quality link or two pointing to your pages. That always speeds things along but rarely happens. The best thing you can do to speed up the process is to promote your site and seek realted/quality links ...


Only top voted, non community-wiki answers of a minimum length are eligible