18

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. The recommended method to ...


15

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature. You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...


12

The first step would be to detect the User-Agent of the bots you want to allow, and serve a different file if it is not a User-Agent that you want to allow. For example, you could have two versions of robots.txt, one with and one without a reference to the sitemap, so your competitors won't find the sitemap if they look inside your robots.txt. Then, you ...


12

MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?) <Files ~ "sitemap.*\.xml(\.gz)?$"> Header append X-Robots-Tag "noindex" </Files> Under nginx ...


10

The value of the xmlns attribute is to uniquely identify a namespace (the "namespace URI"). It does not relate to whether your site is serving content over HTTP or HTTPS - for this it makes no difference. The sitemaps.org site defines the protocol using http://www.sitemaps.org/schemas/sitemap/0.9 as the recognised namespace - so this should really be the ...


9

Sitemaps not sitemap when dealing with high volume of content Your first mistake would to assume that you use 'a sitemap', when dealing with sites that contain a lot of content you need to think in terms of 'sitemaps'. Google indexes content without a sitemap Now it's worth mentioning that Google will pretty much index any important URL without the need ...


8

Yes, sitemap.xml.gzis a valid name. The .gz extension just means that it's been compressed (using gzip compression), so that it's smaller and served faster. Most search engine bots can read gzip'd compressed content. You can edit the sitemap's XML content just by decompressing sitemap.xml.gz using a compression utility like 7-zip. "Robots.txt" is ...


7

Google periodically checks your submitted sitemap.xml file for updates. You only need to submit it once. See: How often does GWT check dynamic sitemaps?


7

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag ...


7

Yes. Everything you ask for is possible. And here's an example XML sitemap file generated by the Drupal XML sitemap module with a little bit of configuration. Everything is done for you out of the box. http://softkube.com/sitemap.xml If you check the code of the XML file there's a link to an XSL and inside that file you can see the code with references to ...


7

As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap limit, not a website limit. (The file must also be no larger than 50MB*1 (uncompressed) - so whichever comes first.) (*1 Previously 10MB.) Then you can also have ...


6

You are asking two questions here. Does a sitemap need to be XML? The simple answer is no, it doesn't have to be XML. It can be XML file, a text file or RSS/Atom feed (which is basically XML), HTML Sitemap HTML Sitemaps: These are used on your website to display the layout in layers on your website to any customer that would wish too (don't know why they ...


6

The problem is that if you (quite rightly) want your content to be indexed by search engines, anyone who performs a site: search in one of the search engines will be able to see what URLs are indexed. If you want to "hide" your sitemap you could have it on a URL with a "secret" name so it's not obvious to anyone who may be looking for it, but seeing as it'...


6

You need to disable and reenable the sitemap, going to SEO > Sitemap XML. It probably will solve your problem (Yoast bug).


6

You can validate your sitemap on https://www.google.com/webmasters/tools. It will give an error when you use https to refer to the schema. I have also seen other sitemap validators that refuse to validate the https version of the namespace, so using http might be the only correct option, even when your site only uses https.


6

As others have stated, a sitemap isn't necessary for such a small amount of content. If you really want to have one, then including everything on one sitemap is okay. Can you combine an XML Sitemap with an Image XML Sitemap? You can use a separate sitemap to list images, or you can add image information to an existing sitemap. Can you combine an XML ...


5

No, neither is required. By default your site will be indexed by Google. The robots.txt file is useful for preventing it from accessing specific directories or files. It's not a security mechanism, however, and if you don't want the public to access those pages/files, you should block access another way, such as an .htaccess file. The sitemap.xml is also ...


5

Search engines will crawl your site more frequently if your content changes more often. If you're just adding new URL's, then there's no need to regenerate your sitemap.xml each time. However, you should update the last modification date in your sitemap by modifying the lastmod attribute, and specify the changefreq for any URL's that will periodically ...


5

The crafty solution is to generate two sitemaps. The first of these is for the benefit of your competitors the second is for the benefit of your preferred search engines. In military parlance this first sitemap is a feint. The 'feint' contains your basic website structure, home page, contact us, about us, main categories. It looks like the real deal and ...


5

You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines. If you have property/form URLs in your sitemap, you've probably used a 3rd party Sitemap Generator. These just crawl all links on your website indescriminately, whereas something like https://www....


5

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to identify content on the site but even if the lastmod tag shows the content hasn't been updated since the last crawl the crawler will still crawl it to confirm the ...


5

Not long ago I was in the same boat when using WordPress plugin Yoast SEO it produced several sitemaps and I was curious to know which one to load to Google's Webmaster Tools Search Console I ran across this article on Google regarding Search Console Help: Simplify multiple sitemap management If you have many sitemaps, you can use a sitemaps index file as ...


4

Yes, change them to the new URLs. Although the 301 redirects tell the search engines where to find moved pages you shouldn't be feeding them inaccurate information about your web pages. Besides possibly being error prone, it may also be an indicator of quality. While this may not affect your rankings, it may possibly affect crawl rate and other related ...


4

2013-07-06T09:39:51-0400 The W3C datetime format includes a : (colon) in the time zone offset, separating the hours and minutes. So the last bit should read: -04:00.


4

It has been verified that a PHP file can be used as a sitemap file such as sitemap.php I checked http://sitemaps.org (the website with the specification information for sitemaps) and there is no mention that the file must be saved in the .XML format. I will have to say that yes, it will work, as long as Google does not have a problem with the extension for ...


4

First of all, the number of results shown by a site: search is only an estimate. Try going forward to the next page in the site:mydomain.com search results, and see if the number changes, or if you find something revealing (like some files that shouldn't be there). If there's really an issue, it could be duplicate content, or some non-HTML content that ...


4

a) just increase the number of categories as soon as google got almost everything crawled? b) just take out categories that do not work and put in others that work better? So that Google Bot is focused on our top 50 categories (with approx 250.000 Products) Not placing pages in your sitemap doesn't hide them from Google as they can find them by naturally ...


4

Outside of signing up for the various web master tools accounts, Bing, Google, Baidu, et. al., the robots.txt file can be used to generally announce the existence of a sitemap file. You use the following example in your robots.txt file which is possibly the best way to get the various search engines (generally speaking) to use your sitemap. Sitemap: http://...


4

Yeah. I have seen this too. You are not alone. I don't see any reason to do this. In fact it is redundant isn't it? And since sitemaps are designed to inform search engine about resources to be indexed and sitemaps are not indexed, it seems rather pointless. It was always a silly thing to do. You will see silly things all over the place especially in ...


4

There are no SEO benefits to serving a compressed XML sitemap over serving one that is not compressed. The advantage of compression is simply to save bandwidth and the time it takes to download. (If your sitemap is huge.) Note that the limits for the size of the sitemap are the uncompressed size (ie. 50MB uncompressed for Google).


Only top voted, non community-wiki answers of a minimum length are eligible