28
You have a misconception of what a sitemap is.
The sitemap is used to audit the site crawl by the search engine bot. The sitemap and crawling a site are two different and independent things. Google will continue to crawl your site independently of any sitemap. The sitemap will be used to audit/see if Google is able to properly crawl your site. For example, ...
18
Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed.
The recommended method to ...
18
Closetnoc is correct about sitemaps. Don't expect them to limit what URLs Google will crawl and index. In fact sitemaps have little to no influence over SEO. See The Sitemap Paradox
Google won't complain about errors from your old URLs if you redirect them. When you change your site's URL structure it is best to redirect all the old URLs to their ...
answered Oct 25 '19 at 1:27
14
Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap.
Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature.
You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...
answered Jun 4 '13 at 0:20
14
In your header you have a canonical link (on line 11, just under <title>).
It looks like this on your page:
<link href="http://escene.ir/component/products/?task=view.12" rel="canonical" />
This element tells Google your preferred URL for a page which has several urls to choose from. This is to prevent you from being penalized for having ...
12
I'm not sure if this changed over the years since this was asked; while in theory you can (as the first answer states), in practice Google at least will give you an error (as seen in their Webmaster Tools):
Incorrect Sitemap index format: Nested Sitemap indexes
The Google help page further states:
A sitemap index file can't list other sitemap index ...
12
You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.
RewriteRule ^robots\.txt$ /robots.php [L]
That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...
answered Oct 22 '15 at 9:23
11
MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this.
Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?)
<Files ~ "sitemap.*\.xml(\.gz)?$">
Header append X-Robots-Tag "noindex"
</Files>
Under nginx ...
answered May 21 '14 at 17:10
11
You can have multiple sitemaps per website, and this is a great example of when that makes sense.
You should make sure you have a Sitemap Index listing each of your sitemaps. It will probably look something like:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
...
11
Generally, you have to use a separate sitemap for each host (i.e., different protocol, domain, or subdomain):
From the FAQ "Where do I place my Sitemap?":
All URLs listed in the Sitemap must reside on the same host as the Sitemap. For instance, if the Sitemap is located at http://www.example.com/sitemap.xml, it can't include URLs from http://...
8
It is not possible to define any hierarchal structure in your XML sitemap. The XML sitemap is a straight forward list of all your pages. Any hierarchal structure to your pages will be determined by Google when it crawls your site.
The hierarchal structure is more relevant to your users. So, your HTML sitemap (if you have one) could be defined in this way. ...
8
First you should address how search engines crawl your site and avoid duplicate content issues by making sure you have the www subdomain (www.example.com) redirecting to your root domain (example.com).
The most efficient way to do this is to create a DNS record - see the first example here on how. You should also use your web server's configuration to ...
7
Google periodically checks your submitted sitemap.xml file for updates. You only need to submit it once.
See: How often does GWT check dynamic sitemaps?
answered Jul 26 '13 at 21:41
7
A single XML sitemap should not contain a mix of HTTP and HTTPS URLs (i.e. essentially different locations as far as the search engines are concerned). So, the sitemap located at http://example.com/Sitemap.xml should only contain URLs starting http://example.com/ and similar for the HTTPS sitemap.
From sitemaps.org:
Q: My site has both "http" and "https" ...
7
The purpose of sitemaps is to tell the search engines about the pages in your website that you want them to crawl and index. If new pages are added to your site that you want crawled and indexed then they should be added to your sitemap. If this is occurring daily then you can add them daily. If this is occurring weekly then you can add them weekly. Search ...
7
The sitemap should reflect the way pages should be accessed, if you want them to be accessed by HTTPS, then yes, that is what you have to do. Otherwise, you are just making the server work a bit extra each time it's crawled.
7
We have another question here that ask why items in the sitemap are not ranked better: The Sitemap Paradox. Google's John Mueller has this to say about common SEO misconceptions regarding sitemaps:
The Sitemap file isn't meant to "fix" crawlability issues. If your site can't be crawled, fix that first.
We don't use Sitemap files for ranking.
...
answered May 29 '14 at 16:23
7
No, you don't need to submit a sitemap for the version (here with www) you don't want to use.
Remember submitting a sitemap to your Google Webmaster Tools account helps indexing of the site. Therefore, you submit a sitemap only for a site you want to see in Google's index.
7
As per Matt Cutts blog post, he said:
A subdomain can be useful to separate out content that is completely
different. Google uses subdomains for distinct products such
news.google.com or maps.google.com, for example.
This is similar to what Blogger (blogspot) does.
John's blog (john.blogspot.com) is totally different from Doe's blog (doe.blogspot....
7
As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap limit, not a website limit. (The file must also be no larger than 50MB*1 (uncompressed) - so whichever comes first.)
(*1 Previously 10MB.)
Then you can also have ...
7
Google will never index HTTPS while the canonicals point to HTTP. I switched my largest site over to HTTPS using the following protocol:
Enabled HTTPS for the site without switching canonicals for about two years. During this time period Google sent all traffic to HTTP.
Switched the canonical version to HTTPS. It has been running that way for about 8 ...
answered Oct 16 '17 at 11:33
7
That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience.
Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, you need to implement the page level redirects
If there is no equivalent for some particular page you can let it 404 and Google will remove it from the index. ...
answered Jan 22 '19 at 13:52
6
This post is 4 years old, hope the status in not Pending anymore. But this helped me.
After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call for my sitemap. Few hours later, my site was indexed.
Upload your sitemap (but you already did hence the pending status)
Typ in your browser: http://www.bing....
6
Your resident SEO expert has valid points, but they're all circumstantial.
Decreased keyword density for the words being targeted in the URL, the
longer the URL the less emphasis is being placed on the keywords being
targeted in the URL
This is an important factor if you have a url like mysite.example/solutions/healthcare/benefits/etc/etc/. But just ...
6
Edit: Google will recrawl it (see comment by John Mueller from Google below).
However, if you want your sitemap reprocessed more quickly by Google, the recommended practice is to resubmit it.
You can resubmit using Webmaster Tools or using an HTTP request.
Using Webmaster Tools:
On the Webmaster Tools Home page, click the site you want.
Under ...
6
The fetching of sitemap files is actually performed by googlebot. The frequency of fetching sitemaps appears to be governed by pagerank and change frequency, similar to how other urls are scheduled for re-crawl.
I have several sites with sitemap ranging in PR from two to five. Googlebot appears to fetch the sitemap files between once a month and six ...
answered Feb 19 '13 at 13:43
6
Why would this be considered harmful? They're perfectly valid URLs that lead to perfectly valid pages. As long as they are available to humans and search engines alike this is fine.
6
You are asking two questions here.
Does a sitemap need to be XML?
The simple answer is no, it doesn't have to be XML. It can be XML
file, a text file or RSS/Atom feed (which is basically XML), HTML
Sitemap
HTML Sitemaps: These are used on your website to display the layout
in layers on your website to any customer that would wish too (don't
know why they ...
6
http://zhanzhang.baidu.com/sitemap/index is the (sparse) documentation about sitemaps for Baidu. It only describes that it may help to submit your sitemap, and it shows a screenshot of their webmaster tool, listing several sitemaps and their status. However, it seems that they intend this feature only for "high quality" sites, on invitation basis.
Here is ...
6
Most likely some part of your web site generated links like that, and that is how Google started to crawl the URLs.
You should check the links in your web pages to see where these incorrect URLs are, and you should fix them.
Also, you could change your Apache configuration so that requests for any other virtualhost than example.com or www.example.com would ...
Only top voted, non community-wiki answers of a minimum length are eligible
