19 votes
Accepted

Prevent XML sitemaps from showing up in Google search results

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I ...
user avatar
  • 41.2k
14 votes

How to hide my XML Sitemap from competitors but not from search engines

The first step would be to detect the User-Agent of the bots you want to allow, and serve a different file if it is not a User-Agent that you want to allow. For example, you could have two versions of ...
user avatar
14 votes

Prevent XML sitemaps from showing up in Google search results

MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - ...
user avatar
12 votes
Accepted

sitemap.xml not rendered as XML document in browser

This suggests your HTTP server is delivering the sitemap with a text/html content type rather than an application/xml one. You need to configure your server correctly.
user avatar
  • 1,580
10 votes

Is incorrect to have the HTTPS version of the sitemaps.org URL in the xmlns sitemap schema?

The value of the xmlns attribute is to uniquely identify a namespace (the "namespace URI"). It does not relate to whether your site is serving content over HTTP or HTTPS - for this it makes no ...
user avatar
  • 4,277
8 votes

Is it possible to make an XML sitemap pretty enough to show to users?

Yes. Everything you ask for is possible. And here's an example XML sitemap file generated by the Drupal XML sitemap module with a little bit of configuration. Everything is done for you out of the box....
user avatar
7 votes
Accepted

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the ...
user avatar
  • 1,424
7 votes

Are alternate links included in the 50,000 URL limit for sitemaps?

As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap ...
user avatar
  • 41.2k
6 votes

How important is it to include <lastmod> in a sitemap?

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to ...
user avatar
6 votes
Accepted

Yoast SEO plugin creates XML Sitemap with http. Is this correct for a https site?

You need to disable and reenable the sitemap, going to SEO > Sitemap XML. It probably will solve your problem (Yoast bug).
user avatar
6 votes
Accepted

Is incorrect to have the HTTPS version of the sitemaps.org URL in the xmlns sitemap schema?

You can validate your sitemap on https://www.google.com/webmasters/tools. It will give an error when you use https to refer to the schema. I have also seen other sitemap validators that refuse to ...
user avatar
  • 76
6 votes

Do I need an XML sitemap for just a few images and videos?

As others have stated, a sitemap isn't necessary for such a small amount of content. If you really want to have one, then including everything on one sitemap is okay. Can you combine an XML Sitemap ...
user avatar
5 votes

How to hide my XML Sitemap from competitors but not from search engines

The crafty solution is to generate two sitemaps. The first of these is for the benefit of your competitors the second is for the benefit of your preferred search engines. In military parlance this ...
user avatar
  • 111
5 votes
Accepted

Can a very large not-so-relevant sitemap hurt my site to Google’s eyes?

You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines. If you have property/form URLs in your ...
user avatar
  • 5,794
5 votes

Is it okay to have multiple sitemaps?

Not long ago I was in the same boat when using WordPress plugin Yoast SEO it produced several sitemaps and I was curious to know which one to load to Google's Webmaster Tools Search Console I ran ...
user avatar
5 votes
Accepted

What does url > lastmod in a sitemap really mean?

I just stumbled across Google's Best practices for XML sitemaps & RSS/Atom feeds, which describes lastmod as follows: The last modification time should be the last time the content of the page ...
user avatar
  • 209
4 votes
Accepted

Where can/should I submit my sitemap.xml?

Outside of signing up for the various web master tools accounts, Bing, Google, Baidu, et. al., the robots.txt file can be used to generally announce the existence of a sitemap file. You use the ...
user avatar
  • 32.5k
4 votes
Accepted

Can XML Sitemap include itself as one of the URLs?

Yeah. I have seen this too. You are not alone. I don't see any reason to do this. In fact it is redundant isn't it? And since sitemaps are designed to inform search engine about resources to be ...
user avatar
  • 32.5k
4 votes
Accepted

Are there any SEO benefits to gzipping sitemap.xml?

There are no SEO benefits to serving a compressed XML sitemap over serving one that is not compressed. The advantage of compression is simply to save bandwidth and the time it takes to download. (If ...
user avatar
  • 41.2k
4 votes
Accepted

Can HTML sitemaps be listed in robots.txt just like XML sitemaps?

Can I list sitemap.php in the robots.txt? No, that wouldn't work. The XML Sitemap goes in robots.txt. The public-facing site map will be crawled by search engines like any other page.
user avatar
  • 7,731
4 votes

Do I need to update the sitemap.xml file after removing some URLs?

It is recommended to keep a sitemap.xml up-to-date and free of errors*. If you merge the content of www.example.com/one-yellow.html and www.example.com/one-yellow-means.html into www.example.com/one....
user avatar
  • 626
4 votes

Should I use HTTPS in sitemap URLs for SEO?

Your sitemap should include the links you want search engines to find and index. If you're providing HTTPS canonicals then this is what should be included in the sitemap. Giving a HTTP in your sitemap ...
user avatar
  • 5,794
4 votes
Accepted

Do you need to put links to pages like "password recovery" to the sitemap?

You should add the URLs of your pages which you want to get crawled and indexed. Don’t you want your password recovery page to get crawled/indexed? Then don’t add it. (And if you want to disallow ...
user avatar
  • 21.5k
4 votes

Is it OK to "pause" an advert in terms of SEO?

1) Remove all links to the ad. If someone manages to come across the ad, then raise an error 404. When the ad resumes, the link becomes available again. I think you need to get feedback on what users ...
user avatar
  • 41.2k
4 votes

How do you get a permanent link for Vimeo hosted videos for use in a video sitemap?

Well it appears you have 3 options. 1. Determine if the token is a temp token and doesn't expire/change and use it. (Most likely it does; there business model is to get the user to use the embedded ...
user avatar
4 votes
Accepted

Why is the sitemap validator returning an error for the "lastmod" tag?

Per the schema, tLastMod is the union of xsd:date and xsd:dateTime. Since you provide hours, you are not in the first case, so your string should conform to the XSD dateTime type, and it does not: ...
user avatar
4 votes
Accepted

How often do search bots typically visit a website?

Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do. Search bots almost ...
user avatar
4 votes

XML Sitemap Couldn't Fetch for about 2 months

This is a known bug of Google new search console. I thought there is some issue in my sitemap, so I have regenerated the sitemap and submitted but it throws the same error. If you look on the google ...
user avatar
  • 141
4 votes
Accepted

Why the new Google Search Console does not read my sitemap?

You may not be doing anything wrong at all. "General HTTP error" can mean many things. The Internet/Web is not a guaranteed delivery protocol. Things will go wrong from time to time. This is not ...
user avatar
  • 32.5k
4 votes
Accepted

Google can't fetch large sitemap with 50k URLs, nor will browsers render it

You should test your sitemap with a downloading program such as curl or wget instead of using a browser like Chrome or Firefox. You should be able to download the file within 3 minutes with a ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible