30 votes
Accepted

Google Search Console is making up URLs which don't exist in my Sitemap and then complains that these pages have error

You have a misconception of what a sitemap is. The sitemap is used to audit the site crawl by the search engine bot. The sitemap and crawling a site are two different and independent things. Google ...
closetnoc's user avatar
  • 32.8k
18 votes

Google Search Console is making up URLs which don't exist in my Sitemap and then complains that these pages have error

Closetnoc is correct about sitemaps. Don't expect them to limit what URLs Google will crawl and index. In fact sitemaps have little to no influence over SEO. See The Sitemap Paradox Google won't ...
Stephen Ostermiller's user avatar
12 votes
Accepted

sitemap.xml not rendered as XML document in browser

This suggests your HTTP server is delivering the sitemap with a text/html content type rather than an application/xml one. You need to configure your server correctly.
Quentin's user avatar
  • 1,608
8 votes

HTTP to HTTPS: Wait for new sitemap to be indexed?

Google will never index HTTPS while the canonicals point to HTTP. I switched my largest site over to HTTPS using the following protocol: Enabled HTTPS for the site without switching canonicals for ...
Stephen Ostermiller's user avatar
8 votes

Why did Yoast put a no-index tag in my XML sitemap?

Because it is an XML file, the sitemap doesn't support meta tags. Instead it is technically an HTTP header: X-Robots-Tag: noindex You don't want to remove it that header. It prevents the XML ...
Stephen Ostermiller's user avatar
7 votes

Are alternate links included in the 50,000 URL limit for sitemaps?

As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap ...
MrWhite's user avatar
  • 42.8k
7 votes
Accepted

Ampersand (&) in actual URL and sitemap

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with &amp; ... ...
MrWhite's user avatar
  • 42.8k
7 votes

Shall I submit sitemap containing all the bad (now 404) URLs after the spam URL hack?

You can submit a temporary sitemap with all the hacked URLs to Google as a method for getting Googlebot to come crawl all of them quickly. When you submit a sitemap of URLs, it will trigger ...
Stephen Ostermiller's user avatar
6 votes
Accepted

How long does Bingbot typically take to process a sitemap submission?

This post is 4 years old, hope the status in not Pending anymore. But this helped me. After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call ...
langeleppel's user avatar
6 votes

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

This will not affect your SEO. Any web crawler will be able to find a website's HTML sitemap as long as there is a link pointing to it. Remember that you can submit your sitemap to most search engines ...
Raul Reyes's user avatar
  • 2,476
6 votes

How important is it to include <lastmod> in a sitemap?

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to ...
Chris Rutherfurd's user avatar
6 votes
Accepted

Yoast SEO plugin creates XML Sitemap with http. Is this correct for a https site?

You need to disable and reenable the sitemap, going to SEO > Sitemap XML. It probably will solve your problem (Yoast bug).
Marcelo Srougi's user avatar
6 votes
Accepted

How to tell bots to forget a website and reindex it from scratch

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
Stephen Ostermiller's user avatar
6 votes

Shall I submit sitemap containing all the bad (now 404) URLs after the spam URL hack?

Leave it as is and hope for the best. Why you shouldn't list 404s in your sitemap.xml Your sitemap should only contain URLs you wan't indexed. URLs in your sitemaps should be clean -they should return ...
Mike Ciffone's user avatar
  • 6,360
5 votes

Does Google penalize daily updated <lastmod> tags in sitemaps if the data is not daily updated?

I've never liked the idea of updating <lastmod> every day as itt's not just wrong, it's misleading search engines. In a post over on SO, Google's Gary Illyes wrote: The lastmod tag is ...
Andrew Lott's user avatar
  • 5,854
5 votes

Two sitemap.xml with wordpress in subfolder

A sitemap is so that Google can index your pages without having to scrape and investigate every internal link. If you want something indexed, it should be in there. If you don't want it indexed - ...
L Martin's user avatar
  • 4,003
5 votes

Should I use HTTPS in sitemap URLs for SEO?

Your sitemap should include the links you want search engines to find and index. If you're providing HTTPS canonicals then this is what should be included in the sitemap. Giving a HTTP in your sitemap ...
Andrew Lott's user avatar
  • 5,854
5 votes
Accepted

Trailing slash, google search console and sitemap

When it comes to the "main domain" (ie. no apparent URL path) there is really only one URL. That is, the one with a trailing slash: //example.com/ However, Google "understands" this and essentially "...
MrWhite's user avatar
  • 42.8k
5 votes

Is it okay to have multiple sitemaps?

Not long ago I was in the same boat when using WordPress plugin Yoast SEO it produced several sitemaps and I was curious to know which one to load to Google's Webmaster Tools Search Console I ran ...
DᴀʀᴛʜVᴀᴅᴇʀ's user avatar
5 votes

Changing sitemap.xml and robots.txt after moving from http to https

The first approach is the correct one. We successfully migrated a high traffic and rankings website from HTTP to HTTPS completely. The approach based on Google guidelines was: All HTTP URLs do a 301 ...
DhruvPathak's user avatar
5 votes
Accepted

How often do search bots typically visit a website?

Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do. Search bots almost ...
Stephen Ostermiller's user avatar
5 votes
Accepted

What is difference between robots.txt, sitemap, robots meta tag, robots header tag?

While X-Robots-Tag and meta robots are equivalent, robots.txt is different. The former is about indexing, while the latter is about crawling/visiting. Tell bots not to visit a URL by using robots.txt....
Maximillian Laumeister's user avatar
5 votes
Accepted

Cache-Control for ads.txt, robots.txt and sitemap.xml files that are generated dynamically

I understand you want to use Cache-Control private/no-store because you don't want the CDN to cache an outdated version of those files. However, not caching these files (i.e. all requests go back to ...
iangetz's user avatar
  • 166
5 votes
Accepted

How should I choose which URLs I want indexed to include in my sitemap?

The pages you most want indexed are your post pages such as /blog/post-1. Those text-heavy pages with article-like content on them are the most attractive to search engines. Collectively, those ...
Stephen Ostermiller's user avatar
5 votes
Accepted

How to make a sitemap with Laravel

saying that a sitemap will affect all the child directories from where the file containing the sitemap is located. I'm not a fan of that wording. It's basically saying that a sitemap can only contain ...
MrWhite's user avatar
  • 42.8k
5 votes
Accepted

Do I still need to ping Google after changing a sitemap nowadays?

According to Google's official Search Central dev docs (under the Submit Your Sitemap to Google section), they still require you to ping them when you change your sitemap to make sure it gets crawled. ...
Trav's user avatar
  • 166
5 votes
Accepted

Cross-domain sitemap submission

It is not a bug. I.e. you are a validated owner of GSC properties of siteA.com and siteB.com. If you are logged in into your GSC property of siteA.com, you can add only sitemap hosted under siteA.com, ...
Evgeniy's user avatar
  • 10.1k
4 votes
Accepted

Moving folder's content to a new domain, what SEO steps to take?

You have all the steps correct, but it won't prevent a temporary (up to 8 month) drop in SEO traffic. Google does not support migrating an entire folder to its own new domain. When you do so, your ...
Stephen Ostermiller's user avatar
4 votes
Accepted

Can HTML sitemaps be listed in robots.txt just like XML sitemaps?

Can I list sitemap.php in the robots.txt? No, that wouldn't work. The XML Sitemap goes in robots.txt. The public-facing site map will be crawled by search engines like any other page.
GDVS's user avatar
  • 7,795

Only top scored, non community-wiki answers of a minimum length are eligible