30
votes
Accepted
Google Search Console is making up URLs which don't exist in my Sitemap and then complains that these pages have error
You have a misconception of what a sitemap is.
The sitemap is used to audit the site crawl by the search engine bot. The sitemap and crawling a site are two different and independent things. Google ...
- 32.6k
18
votes
Google Search Console is making up URLs which don't exist in my Sitemap and then complains that these pages have error
Closetnoc is correct about sitemaps. Don't expect them to limit what URLs Google will crawl and index. In fact sitemaps have little to no influence over SEO. See The Sitemap Paradox
Google won't ...
- 95.4k
14
votes
Accepted
Can a sitemap index contain other sitemap indexes?
I'm not sure if this changed over the years since this was asked; while in theory you can (as the first answer states), in practice Google at least will give you an error (as seen in their Webmaster ...
- 256
12
votes
Accepted
Is a different sitemap per language OK? How do I tell Google about them?
You can have multiple sitemaps per website, and this is a great example of when that makes sense.
You should make sure you have a Sitemap Index listing each of your sitemaps. It will probably look ...
- 5,814
12
votes
Sitemaps - one per subdomain or one for the base domain?
Generally, you have to use a separate sitemap for each host (i.e., different protocol, domain, or subdomain):
From the FAQ "Where do I place my Sitemap?":
All URLs listed in the Sitemap ...
- 21.5k
12
votes
Accepted
Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?
You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.
RewriteRule ^robots\.txt$ /robots.php [L]
That way, you power it with a dynamic script, ...
- 95.4k
12
votes
Accepted
sitemap.xml not rendered as XML document in browser
This suggests your HTTP server is delivering the sitemap with a text/html content type rather than an application/xml one.
You need to configure your server correctly.
- 1,609
9
votes
Accepted
Sitemaps - one per subdomain or one for the base domain?
As per Matt Cutts blog post, he said:
A subdomain can be useful to separate out content that is completely
different. Google uses subdomains for distinct products such
news.google.com or maps....
8
votes
Is it possible to make an XML sitemap pretty enough to show to users?
Yes. Everything you ask for is possible. And here's an example XML sitemap file generated by the Drupal XML sitemap module with a little bit of configuration. Everything is done for you out of the box....
- 431
7
votes
Which is the best choice for SEO sitemap for multilingual website?
You're better off creating a multilingual sitemap, just to avoid any source of confusion. The format you have shown is correct.
In fact, you could even drop the hreflang declarations in your page ...
- 676
7
votes
Are alternate links included in the 50,000 URL limit for sitemaps?
As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap ...
- 41.6k
7
votes
HTTP to HTTPS: Wait for new sitemap to be indexed?
Google will never index HTTPS while the canonicals point to HTTP. I switched my largest site over to HTTPS using the following protocol:
Enabled HTTPS for the site without switching canonicals for ...
- 95.4k
7
votes
Accepted
Ampersand (&) in actual URL and sitemap
if this difference of ampersand in URL and sitemap will cause any issue.
tl;dr No issue, because the URLs are the same.
Since in sitemap & has to be escaped I replaced & with & ...
...
- 41.6k
7
votes
Accepted
How to tell bots to forget a website and reindex it from scratch
That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience.
Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
- 95.4k
7
votes
Why did Yoast put a no-index tag in my XML sitemap?
Because it is an XML file, the sitemap doesn't support meta tags. Instead it is technically an HTTP header:
X-Robots-Tag: noindex
You don't want to remove it that header. It prevents the XML ...
- 95.4k
6
votes
Accepted
How long does Bingbot typically take to process a sitemap submission?
This post is 4 years old, hope the status in not Pending anymore. But this helped me.
After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call ...
- 76
6
votes
Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?
This will not affect your SEO. Any web crawler will be able to find a website's HTML sitemap as long as there is a link pointing to it.
Remember that you can submit your sitemap to most search engines ...
- 2,428
6
votes
How important is it to include <lastmod> in a sitemap?
Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to ...
- 10.1k
6
votes
Accepted
Yoast SEO plugin creates XML Sitemap with http. Is this correct for a https site?
You need to disable and reenable the sitemap, going to SEO > Sitemap XML. It probably will solve your problem (Yoast bug).
- 144
5
votes
Does Google penalize daily updated <lastmod> tags in sitemaps if the data is not daily updated?
I've never liked the idea of updating <lastmod> every day as itt's not just wrong, it's misleading search engines.
In a post over on SO, Google's Gary Illyes wrote:
The lastmod tag is ...
- 5,814
5
votes
Accepted
Can a very large not-so-relevant sitemap hurt my site to Google’s eyes?
You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines.
If you have property/form URLs in your ...
- 5,814
5
votes
Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?
Yes, the same way any request can be "dynamic".
However, you would not redirect (as in your example code), you should internally rewrite using mod_rewrite. (The same as what Drupal is probably ...
- 41.6k
5
votes
Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?
Making the sitemap file dynamic is fine -- it's a good way to auto-update your sitemaps.
Making the robots.txt file dynamic (for the same host! Doing this for separate hosts is essentially just a ...
- 7,371
5
votes
Two sitemap.xml with wordpress in subfolder
A sitemap is so that Google can index your pages without having to scrape and investigate every internal link. If you want something indexed, it should be in there. If you don't want it indexed - ...
- 3,868
5
votes
Should I use HTTPS in sitemap URLs for SEO?
Your sitemap should include the links you want search engines to find and index. If you're providing HTTPS canonicals then this is what should be included in the sitemap. Giving a HTTP in your sitemap ...
- 5,814
5
votes
Accepted
Trailing slash, google search console and sitemap
When it comes to the "main domain" (ie. no apparent URL path) there is really only one URL. That is, the one with a trailing slash:
//example.com/
However, Google "understands" this and essentially "...
- 41.6k
5
votes
Is it okay to have multiple sitemaps?
Not long ago I was in the same boat when using WordPress plugin Yoast SEO it produced several sitemaps and I was curious to know which one to load to Google's Webmaster Tools Search Console I ran ...
- 671
5
votes
Changing sitemap.xml and robots.txt after moving from http to https
The first approach is the correct one. We successfully migrated a high traffic and rankings website from HTTP to HTTPS completely.
The approach based on Google guidelines was:
All HTTP URLs do a 301 ...
- 221
5
votes
Accepted
What is difference between robots.txt, sitemap, robots meta tag, robots header tag?
While X-Robots-Tag and meta robots are equivalent, robots.txt is different. The former is about indexing, while the latter is about crawling/visiting.
Tell bots not to visit a URL by using robots.txt....
- 14.9k
5
votes
Accepted
How should I choose which URLs I want indexed to include in my sitemap?
The pages you most want indexed are your post pages such as /blog/post-1. Those text-heavy pages with article-like content on them are the most attractive to search engines. Collectively, those ...
- 95.4k
Only top scored, non community-wiki answers of a minimum length are eligible