28
You have a misconception of what a sitemap is.
The sitemap is used to audit the site crawl by the search engine bot. The sitemap and crawling a site are two different and independent things. Google will continue to crawl your site independently of any sitemap. The sitemap will be used to audit/see if Google is able to properly crawl your site. For example, ...
19
Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed.
The recommended method to ...
18
Closetnoc is correct about sitemaps. Don't expect them to limit what URLs Google will crawl and index. In fact sitemaps have little to no influence over SEO. See The Sitemap Paradox
Google won't complain about errors from your old URLs if you redirect them. When you change your site's URL structure it is best to redirect all the old URLs to their ...
answered Oct 25 '19 at 1:27
14
Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap.
Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature.
You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...
answered Jun 4 '13 at 0:20
14
In your header you have a canonical link (on line 11, just under <title>).
It looks like this on your page:
<link href="http://escene.ir/component/products/?task=view.12" rel="canonical" />
This element tells Google your preferred URL for a page which has several urls to choose from. This is to prevent you from being penalized for having ...
12
I'm not sure if this changed over the years since this was asked; while in theory you can (as the first answer states), in practice Google at least will give you an error (as seen in their Webmaster Tools):
Incorrect Sitemap index format: Nested Sitemap indexes
The Google help page further states:
A sitemap index file can't list other sitemap index ...
12
MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this.
Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?)
<Files ~ "sitemap.*\.xml(\.gz)?$">
Header append X-Robots-Tag "noindex"
</Files>
Under nginx ...
answered May 21 '14 at 17:10
12
You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.
RewriteRule ^robots\.txt$ /robots.php [L]
That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...
answered Oct 22 '15 at 9:23
11
You can have multiple sitemaps per website, and this is a great example of when that makes sense.
You should make sure you have a Sitemap Index listing each of your sitemaps. It will probably look something like:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
...
11
Generally, you have to use a separate sitemap for each host (i.e., different protocol, domain, or subdomain):
From the FAQ "Where do I place my Sitemap?":
All URLs listed in the Sitemap must reside on the same host as the Sitemap. For instance, if the Sitemap is located at http://www.example.com/sitemap.xml, it can't include URLs from http://...
8
First you should address how search engines crawl your site and avoid duplicate content issues by making sure you have the www subdomain (www.example.com) redirecting to your root domain (example.com).
The most efficient way to do this is to create a DNS record - see the first example here on how. You should also use your web server's configuration to ...
7
Google periodically checks your submitted sitemap.xml file for updates. You only need to submit it once.
See: How often does GWT check dynamic sitemaps?
answered Jul 26 '13 at 21:41
7
A single XML sitemap should not contain a mix of HTTP and HTTPS URLs (i.e. essentially different locations as far as the search engines are concerned). So, the sitemap located at http://example.com/Sitemap.xml should only contain URLs starting http://example.com/ and similar for the HTTPS sitemap.
From sitemaps.org:
Q: My site has both "http" and "https" ...
7
The purpose of sitemaps is to tell the search engines about the pages in your website that you want them to crawl and index. If new pages are added to your site that you want crawled and indexed then they should be added to your sitemap. If this is occurring daily then you can add them daily. If this is occurring weekly then you can add them weekly. Search ...
7
The sitemap should reflect the way pages should be accessed, if you want them to be accessed by HTTPS, then yes, that is what you have to do. Otherwise, you are just making the server work a bit extra each time it's crawled.
7
We have another question here that ask why items in the sitemap are not ranked better: The Sitemap Paradox. Google's John Mueller has this to say about common SEO misconceptions regarding sitemaps:
The Sitemap file isn't meant to "fix" crawlability issues. If your site can't be crawled, fix that first.
We don't use Sitemap files for ranking.
...
answered May 29 '14 at 16:23
7
No, you don't need to submit a sitemap for the version (here with www) you don't want to use.
Remember submitting a sitemap to your Google Webmaster Tools account helps indexing of the site. Therefore, you submit a sitemap only for a site you want to see in Google's index.
7
As per Matt Cutts blog post, he said:
A subdomain can be useful to separate out content that is completely
different. Google uses subdomains for distinct products such
news.google.com or maps.google.com, for example.
This is similar to what Blogger (blogspot) does.
John's blog (john.blogspot.com) is totally different from Doe's blog (doe.blogspot....
7
As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap limit, not a website limit. (The file must also be no larger than 50MB*1 (uncompressed) - so whichever comes first.)
(*1 Previously 10MB.)
Then you can also have ...
7
Google will never index HTTPS while the canonicals point to HTTP. I switched my largest site over to HTTPS using the following protocol:
Enabled HTTPS for the site without switching canonicals for about two years. During this time period Google sent all traffic to HTTP.
Switched the canonical version to HTTPS. It has been running that way for about 8 ...
answered Oct 16 '17 at 11:33
7
if this difference of ampersand in URL and sitemap will cause any issue.
tl;dr No issue, because the URLs are the same.
Since in sitemap & has to be escaped I replaced & with & ...
Your sitemap is an XML document. As with any XML document, the data values must be stored XML-entity encoded. The & character is a special character (it ...
7
That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience.
Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, you need to implement the page level redirects
If there is no equivalent for some particular page you can let it 404 and Google will remove it from the index. ...
answered Jan 22 '19 at 13:52
6
This post is 4 years old, hope the status in not Pending anymore. But this helped me.
After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call for my sitemap. Few hours later, my site was indexed.
Upload your sitemap (but you already did hence the pending status)
Typ in your browser: http://www.bing....
6
Edit: Google will recrawl it (see comment by John Mueller from Google below).
However, if you want your sitemap reprocessed more quickly by Google, the recommended practice is to resubmit it.
You can resubmit using Webmaster Tools or using an HTTP request.
Using Webmaster Tools:
On the Webmaster Tools Home page, click the site you want.
Under ...
6
The fetching of sitemap files is actually performed by googlebot. The frequency of fetching sitemaps appears to be governed by pagerank and change frequency, similar to how other urls are scheduled for re-crawl.
I have several sites with sitemap ranging in PR from two to five. Googlebot appears to fetch the sitemap files between once a month and six ...
answered Feb 19 '13 at 13:43
6
Sitemap.xml helps your website to be well and fast indexed. By submitting this file to search engines, I don't see any good reason to ask them to index a page that doesn't exist (404 HTTP status).
6
Why would this be considered harmful? They're perfectly valid URLs that lead to perfectly valid pages. As long as they are available to humans and search engines alike this is fine.
6
You are asking two questions here.
Does a sitemap need to be XML?
The simple answer is no, it doesn't have to be XML. It can be XML
file, a text file or RSS/Atom feed (which is basically XML), HTML
Sitemap
HTML Sitemaps: These are used on your website to display the layout
in layers on your website to any customer that would wish too (don't
know why they ...
6
http://zhanzhang.baidu.com/sitemap/index is the (sparse) documentation about sitemaps for Baidu. It only describes that it may help to submit your sitemap, and it shows a screenshot of their webmaster tool, listing several sitemaps and their status. However, it seems that they intend this feature only for "high quality" sites, on invitation basis.
Here is ...
6
Most likely some part of your web site generated links like that, and that is how Google started to crawl the URLs.
You should check the links in your web pages to see where these incorrect URLs are, and you should fix them.
Also, you could change your Apache configuration so that requests for any other virtualhost than example.com or www.example.com would ...
Only top voted, non community-wiki answers of a minimum length are eligible
