19

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. The recommended method to ...


14

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature. You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...


12

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


12

MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?) <Files ~ "sitemap.*\.xml(\.gz)?$"> Header append X-Robots-Tag "noindex" </Files> Under nginx ...


10

The first step would be to detect the User-Agent of the bots you want to allow, and serve a different file if it is not a User-Agent that you want to allow. For example, you could have two versions of robots.txt, one with and one without a reference to the sitemap, so your competitors won't find the sitemap if they look inside your robots.txt. Then, you ...


10

The value of the xmlns attribute is to uniquely identify a namespace (the "namespace URI"). It does not relate to whether your site is serving content over HTTP or HTTPS - for this it makes no difference. The sitemaps.org site defines the protocol using http://www.sitemaps.org/schemas/sitemap/0.9 as the recognised namespace - so this should really be the ...


9

Sitemaps not sitemap when dealing with high volume of content Your first mistake would to assume that you use 'a sitemap', when dealing with sites that contain a lot of content you need to think in terms of 'sitemaps'. Google indexes content without a sitemap Now it's worth mentioning that Google will pretty much index any important URL without the need ...


8

The only place on your site where you might place a link (or URL reference) to your XML sitemap is in your robots.txt file. This will allow Google and all the other search engines you've not explicitly submitted a sitemap to, that support this extended robots.txt syntax, to find your XML sitemap: Sitemap: http://example.com/Sitemap.xml Absolute or relative ...


7

Yes, sitemap.xml.gzis a valid name. The .gz extension just means that it's been compressed (using gzip compression), so that it's smaller and served faster. Most search engine bots can read gzip'd compressed content. You can edit the sitemap's XML content just by decompressing sitemap.xml.gz using a compression utility like 7-zip. "Robots.txt" is ...


7

Google periodically checks your submitted sitemap.xml file for updates. You only need to submit it once. See: How often does GWT check dynamic sitemaps?


7

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag ...


7

As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap limit, not a website limit. (The file must also be no larger than 50MB*1 (uncompressed) - so whichever comes first.) (*1 Previously 10MB.) Then you can also have ...


6

No, neither is required. By default your site will be indexed by Google. The robots.txt file is useful for preventing it from accessing specific directories or files. It's not a security mechanism, however, and if you don't want the public to access those pages/files, you should block access another way, such as an .htaccess file. The sitemap.xml is also ...


6

You are asking two questions here. Does a sitemap need to be XML? The simple answer is no, it doesn't have to be XML. It can be XML file, a text file or RSS/Atom feed (which is basically XML), HTML Sitemap HTML Sitemaps: These are used on your website to display the layout in layers on your website to any customer that would wish too (don't know why they ...


6

The problem is that if you (quite rightly) want your content to be indexed by search engines, anyone who performs a site: search in one of the search engines will be able to see what URLs are indexed. If you want to "hide" your sitemap you could have it on a URL with a "secret" name so it's not obvious to anyone who may be looking for it, but seeing as it'...


6

Yes. Everything you ask for is possible. And here's an example XML sitemap file generated by the Drupal XML sitemap module with a little bit of configuration. Everything is done for you out of the box. http://softkube.com/sitemap.xml If you check the code of the XML file there's a link to an XSL and inside that file you can see the code with references to ...


6

As others have stated, a sitemap isn't necessary for such a small amount of content. If you really want to have one, then including everything on one sitemap is okay. Can you combine an XML Sitemap with an Image XML Sitemap? You can use a separate sitemap to list images, or you can add image information to an existing sitemap. Can you combine an XML ...


5

There is no fundamental disadvantage to text-only sitemaps aside from what you've touched on. Setting the priority and last-modified date can affect the crawl frequency and on a large site with both frequently and infrequently changing content that can be very important. XML sitemaps can also carry media information (e.g. video sitemap) and if you have ...


5

Search engines will crawl your site more frequently if your content changes more often. If you're just adding new URL's, then there's no need to regenerate your sitemap.xml each time. However, you should update the last modification date in your sitemap by modifying the lastmod attribute, and specify the changefreq for any URL's that will periodically ...


5

The crafty solution is to generate two sitemaps. The first of these is for the benefit of your competitors the second is for the benefit of your preferred search engines. In military parlance this first sitemap is a feint. The 'feint' contains your basic website structure, home page, contact us, about us, main categories. It looks like the real deal and ...


5

You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines. If you have property/form URLs in your sitemap, you've probably used a 3rd party Sitemap Generator. These just crawl all links on your website indescriminately, whereas something like https://www....


5

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to identify content on the site but even if the lastmod tag shows the content hasn't been updated since the last crawl the crawler will still crawl it to confirm the ...


5

Not long ago I was in the same boat when using WordPress plugin Yoast SEO it produced several sitemaps and I was curious to know which one to load to Google's Webmaster Tools Search Console I ran across this article on Google regarding Search Console Help: Simplify multiple sitemap management If you have many sitemaps, you can use a sitemaps index file as ...


5

You need to disable and reenable the sitemap, going to SEO > Sitemap XML. It probably will solve your problem (Yoast bug).


5

You can validate your sitemap on https://www.google.com/webmasters/tools. It will give an error when you use https to refer to the schema. I have also seen other sitemap validators that refuse to validate the https version of the namespace, so using http might be the only correct option, even when your site only uses https.


4

First of all, the number of results shown by a site: search is only an estimate. Try going forward to the next page in the site:mydomain.com search results, and see if the number changes, or if you find something revealing (like some files that shouldn't be there). If there's really an issue, it could be duplicate content, or some non-HTML content that ...


4

No you shouldn't include them in a regular sitemap, as they are not web pages. Even so, they are essentially just links to actual pages in your site. However, it should be fine to include them in a sitemap index file, because Google recognises RSS feeds as sitemaps.


4

Don't include RSS feed urls in your sitemap. RSS feeds don't make good landing pages. I've never seen any evidence that Google wants to index them. As such, they are not a good fit for sitemap inclusion. They will clutter up the webmaster tools stats of the pages that you do care about. They won't provide any benefit to Google.


4

The XML sitemap really is for search engines (ie, Google) and not humans. Put it on your root and give Google the path to it in your Google Webmaster account, as you did. Make sure you keep it updated, since that really is the benefit of an XML Sitemap. I keep mine updated with a custom cron job that runs a PHP script to generate the new XML Sitemaps every ...


4

2013-07-06T09:39:51-0400 The W3C datetime format includes a : (colon) in the time zone offset, separating the hours and minutes. So the last bit should read: -04:00.


Only top voted, non community-wiki answers of a minimum length are eligible