19

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. The recommended method to ...


14

MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?) <Files ~ "sitemap.*\.xml(\.gz)?$"> Header append X-Robots-Tag "noindex" </Files> Under nginx ...


13

The first step would be to detect the User-Agent of the bots you want to allow, and serve a different file if it is not a User-Agent that you want to allow. For example, you could have two versions of robots.txt, one with and one without a reference to the sitemap, so your competitors won't find the sitemap if they look inside your robots.txt. Then, you ...


10

The value of the xmlns attribute is to uniquely identify a namespace (the "namespace URI"). It does not relate to whether your site is serving content over HTTP or HTTPS - for this it makes no difference. The sitemaps.org site defines the protocol using http://www.sitemaps.org/schemas/sitemap/0.9 as the recognised namespace - so this should really be the ...


9

Sitemaps not sitemap when dealing with high volume of content Your first mistake would to assume that you use 'a sitemap', when dealing with sites that contain a lot of content you need to think in terms of 'sitemaps'. Google indexes content without a sitemap Now it's worth mentioning that Google will pretty much index any important URL without the need ...


7

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag ...


7

Yes. Everything you ask for is possible. And here's an example XML sitemap file generated by the Drupal XML sitemap module with a little bit of configuration. Everything is done for you out of the box. http://softkube.com/sitemap.xml If you check the code of the XML file there's a link to an XSL and inside that file you can see the code with references to ...


7

As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap limit, not a website limit. (The file must also be no larger than 50MB*1 (uncompressed) - so whichever comes first.) (*1 Previously 10MB.) Then you can also have ...


6

You are asking two questions here. Does a sitemap need to be XML? The simple answer is no, it doesn't have to be XML. It can be XML file, a text file or RSS/Atom feed (which is basically XML), HTML Sitemap HTML Sitemaps: These are used on your website to display the layout in layers on your website to any customer that would wish too (don't know why they ...


6

The problem is that if you (quite rightly) want your content to be indexed by search engines, anyone who performs a site: search in one of the search engines will be able to see what URLs are indexed. If you want to "hide" your sitemap you could have it on a URL with a "secret" name so it's not obvious to anyone who may be looking for it, but seeing as it'...


6

You need to disable and reenable the sitemap, going to SEO > Sitemap XML. It probably will solve your problem (Yoast bug).


6

You can validate your sitemap on https://www.google.com/webmasters/tools. It will give an error when you use https to refer to the schema. I have also seen other sitemap validators that refuse to validate the https version of the namespace, so using http might be the only correct option, even when your site only uses https.


6

As others have stated, a sitemap isn't necessary for such a small amount of content. If you really want to have one, then including everything on one sitemap is okay. Can you combine an XML Sitemap with an Image XML Sitemap? You can use a separate sitemap to list images, or you can add image information to an existing sitemap. Can you combine an XML ...


5

The crafty solution is to generate two sitemaps. The first of these is for the benefit of your competitors the second is for the benefit of your preferred search engines. In military parlance this first sitemap is a feint. The 'feint' contains your basic website structure, home page, contact us, about us, main categories. It looks like the real deal and ...


5

You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines. If you have property/form URLs in your sitemap, you've probably used a 3rd party Sitemap Generator. These just crawl all links on your website indescriminately, whereas something like https://www....


5

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to identify content on the site but even if the lastmod tag shows the content hasn't been updated since the last crawl the crawler will still crawl it to confirm the ...


5

Not long ago I was in the same boat when using WordPress plugin Yoast SEO it produced several sitemaps and I was curious to know which one to load to Google's Webmaster Tools Search Console I ran across this article on Google regarding Search Console Help: Simplify multiple sitemap management If you have many sitemaps, you can use a sitemaps index file as ...


4

a) just increase the number of categories as soon as google got almost everything crawled? b) just take out categories that do not work and put in others that work better? So that Google Bot is focused on our top 50 categories (with approx 250.000 Products) Not placing pages in your sitemap doesn't hide them from Google as they can find them by naturally ...


4

Outside of signing up for the various web master tools accounts, Bing, Google, Baidu, et. al., the robots.txt file can be used to generally announce the existence of a sitemap file. You use the following example in your robots.txt file which is possibly the best way to get the various search engines (generally speaking) to use your sitemap. Sitemap: http://...


4

Yeah. I have seen this too. You are not alone. I don't see any reason to do this. In fact it is redundant isn't it? And since sitemaps are designed to inform search engine about resources to be indexed and sitemaps are not indexed, it seems rather pointless. It was always a silly thing to do. You will see silly things all over the place especially in ...


4

There are no SEO benefits to serving a compressed XML sitemap over serving one that is not compressed. The advantage of compression is simply to save bandwidth and the time it takes to download. (If your sitemap is huge.) Note that the limits for the size of the sitemap are the uncompressed size (ie. 50MB uncompressed for Google).


4

Can I list sitemap.php in the robots.txt? No, that wouldn't work. The XML Sitemap goes in robots.txt. The public-facing site map will be crawled by search engines like any other page.


4

It is recommended to keep a sitemap.xml up-to-date and free of errors*. If you merge the content of www.example.com/one-yellow.html and www.example.com/one-yellow-means.html into www.example.com/one.html you probably (ideally) redirect them both to www.example.com/one.html. Other URLs that have been removed may serve an HTTP status 404 (not foud) or ...


4

Your sitemap should include the links you want search engines to find and index. If you're providing HTTPS canonicals then this is what should be included in the sitemap. Giving a HTTP in your sitemap and a HTTPS in your canonicals & internal links could be misleading to search engines. What you shouldn't do is include both. sitemaps.org answers that ...


4

You should add the URLs of your pages which you want to get crawled and indexed. Don’t you want your password recovery page to get crawled/indexed? Then don’t add it. (And if you want to disallow crawling of your password recovery page, add it to robots.txt; if you want to disallow indexing of your password recovery page, use noindex; don’t use both). ...


4

1) Remove all links to the ad. If someone manages to come across the ad, then raise an error 404. When the ad resumes, the link becomes available again. I think you need to get feedback on what users expect to happen when an advert is "paused". This might be what they expect? Although this is essentially "unpublishing" the ad, albeit temporarily. The 404 ...


4

I just stumbled across Google's Best practices for XML sitemaps & RSS/Atom feeds, which describes lastmod as follows: The last modification time should be the last time the content of the page changed meaningfully. If a change is meant to be visible in the search results, then the last modification time should be the time of this change. ...


4

Per the schema, tLastMod is the union of xsd:date and xsd:dateTime. Since you provide hours, you are not in the first case, so your string should conform to the XSD dateTime type, and it does not: you need to specify the part with seconds. See https://www.w3.org/TR/2012/REC-xmlschema11-2-20120405/#dateTime-lexical-mapping for the full format. So, if you ...


4

Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do. Search bots almost never crawl the entire site in one sitting. Rather they re-crawl pages each on their own schedule. They schedule page re-crawls based on how popular a page ...


4

You should test your sitemap with a downloading program such as curl or wget instead of using a browser like Chrome or Firefox. You should be able to download the file within 3 minutes with a download program. If the file takes longer to download for you, then Googlebot will probably also have problems with it. You can: Upgrade your hosting so your ...


Only top voted, non community-wiki answers of a minimum length are eligible