93
A Sitemap file helps search engines to discover new and updated URLs on your website. In particular, if your website is fairly large, then this can help them to be able to focus on the new & updated content, instead of having to blindly crawl through everything to see if anything has changed. That can result in new content being found much faster, which ...
28
You have a misconception of what a sitemap is.
The sitemap is used to audit the site crawl by the search engine bot. The sitemap and crawling a site are two different and independent things. Google will continue to crawl your site independently of any sitemap. The sitemap will be used to audit/see if Google is able to properly crawl your site. For example, ...
18
Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed.
The recommended method to ...
18
Closetnoc is correct about sitemaps. Don't expect them to limit what URLs Google will crawl and index. In fact sitemaps have little to no influence over SEO. See The Sitemap Paradox
Google won't complain about errors from your old URLs if you redirect them. When you change your site's URL structure it is best to redirect all the old URLs to their ...
answered Oct 25 '19 at 1:27
14
Yes and No. From the sitemap.org official site:-
The location of a Sitemap file determines the set of URLs that can be
included in that Sitemap. A Sitemap file located at
http://example.com/catalog/sitemap.xml can include any URLs starting
with http://example.com/catalog/ but can not include URLs starting
with http://example.com/images/.
So yes, ...
14
In your header you have a canonical link (on line 11, just under <title>).
It looks like this on your page:
<link href="http://escene.ir/component/products/?task=view.12" rel="canonical" />
This element tells Google your preferred URL for a page which has several urls to choose from. This is to prevent you from being penalized for having ...
12
I'm not sure if this changed over the years since this was asked; while in theory you can (as the first answer states), in practice Google at least will give you an error (as seen in their Webmaster Tools):
Incorrect Sitemap index format: Nested Sitemap indexes
The Google help page further states:
A sitemap index file can't list other sitemap index ...
12
You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.
RewriteRule ^robots\.txt$ /robots.php [L]
That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...
answered Oct 22 '15 at 9:23
11
Yes, you can do this, just make sure you specify the file's location in your robots.txt file.
I've had to do this for a site where I had limited access to the file structure, so I know it works - looks like other people have tried too.
11
We now utilize a crawling method of adding new content to our search index. The subdomain you refer to was just retired over the weekend, and we'll be updating that FAQ page very shortly.
Best regards,
Eric from Ask.com
11
Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap.
Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature.
You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...
answered Jun 4 '13 at 0:20
11
Generally, you have to use a separate sitemap for each host (i.e., different protocol, domain, or subdomain):
From the FAQ "Where do I place my Sitemap?":
All URLs listed in the Sitemap must reside on the same host as the Sitemap. For instance, if the Sitemap is located at http://www.example.com/sitemap.xml, it can't include URLs from http://subdomain....
10
You can have multiple sitemaps per website, and this is a great example of when that makes sense.
You should make sure you have a Sitemap Index listing each of your sitemaps. It will probably look something like:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
...
9
In XML the ampersand has to be escaped, so change it to: &
9
MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this.
Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?)
<Files ~ "sitemap.*\.xml(\.gz)?$">
Header append X-Robots-Tag "noindex"
</Files>
Under nginx ...
answered May 21 '14 at 17:10
8
It is not possible to define any hierarchal structure in your XML sitemap. The XML sitemap is a straight forward list of all your pages. Any hierarchal structure to your pages will be determined by Google when it crawls your site.
The hierarchal structure is more relevant to your users. So, your HTML sitemap (if you have one) could be defined in this way. ...
8
First you should address how search engines crawl your site and avoid duplicate content issues by making sure you have the www subdomain (www.example.com) redirecting to your root domain (example.com).
The most efficient way to do this is to create a DNS record - see the first example here on how. You should also use your web server's configuration to ...
7
Google periodically checks your submitted sitemap.xml file for updates. You only need to submit it once.
See: How often does GWT check dynamic sitemaps?
answered Jul 26 '13 at 21:41
7
A single XML sitemap should not contain a mix of HTTP and HTTPS URLs (i.e. essentially different locations as far as the search engines are concerned). So, the sitemap located at http://example.com/Sitemap.xml should only contain URLs starting http://example.com/ and similar for the HTTPS sitemap.
From sitemaps.org:
Q: My site has both "http" and "https" ...
7
The purpose of sitemaps is to tell the search engines about the pages in your website that you want them to crawl and index. If new pages are added to your site that you want crawled and indexed then they should be added to your sitemap. If this is occurring daily then you can add them daily. If this is occurring weekly then you can add them weekly. Search ...
7
The sitemap should reflect the way pages should be accessed, if you want them to be accessed by HTTPS, then yes, that is what you have to do. Otherwise, you are just making the server work a bit extra each time it's crawled.
7
We have another question here that ask why items in the sitemap are not ranked better: The Sitemap Paradox. Google's John Mueller has this to say about common SEO misconceptions regarding sitemaps:
The Sitemap file isn't meant to "fix" crawlability issues. If your site can't be crawled, fix that first.
We don't use Sitemap files for ranking.
...
answered May 29 '14 at 16:23
7
No, you don't need to submit a sitemap for the version (here with www) you don't want to use.
Remember submitting a sitemap to your Google Webmaster Tools account helps indexing of the site. Therefore, you submit a sitemap only for a site you want to see in Google's index.
7
As per Matt Cutts blog post, he said:
A subdomain can be useful to separate out content that is completely
different. Google uses subdomains for distinct products such
news.google.com or maps.google.com, for example.
This is similar to what Blogger (blogspot) does.
John's blog (john.blogspot.com) is totally different from Doe's blog (doe.blogspot....
7
As closetnoc suggests in comments, the 50,000 URL limit for sitemaps refers to the number of URLs in the sitemap file itself. ie. the number of <loc> elements. This is an individual sitemap limit, not a website limit. (The file must also be no larger than 50MB*1 (uncompressed) - so whichever comes first.)
(*1 Previously 10MB.)
Then you can also have ...
7
Google will never index HTTPS while the canonicals point to HTTP. I switched my largest site over to HTTPS using the following protocol:
Enabled HTTPS for the site without switching canonicals for about two years. During this time period Google sent all traffic to HTTP.
Switched the canonical version to HTTPS. It has been running that way for about 8 ...
answered Oct 16 '17 at 11:33
7
That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience.
Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, you need to implement the page level redirects
If there is no equivalent for some particular page you can let it 404 and Google will remove it from the index. ...
answered Jan 22 '19 at 13:52
6
RSS helps users browse your site in an aggregated format without having to visit the site. There is some evidence to suggest that feeds help keep your site 'freshness' up in search engine rankings so they are worth the effort, but how much this affects your ranking is anyone's guess.
Sitemaps helps search engines index your site and better understand when ...
6
If you're not getting any errors then you can assume Google has parsed it and is aware of the contents. But that doesn't mean they will crawl and/or index those pages. Sitemaps are just another way to tell search engines about your pages. They are not obligated to crawl and index any or all of those pages. The same applies to them finding pages through links ...
6
This post is 4 years old, hope the status in not Pending anymore. But this helped me.
After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call for my sitemap. Few hours later, my site was indexed.
Upload your sitemap (but you already did hence the pending status)
Typ in your browser: http://www.bing....
Only top voted, non community-wiki answers of a minimum length are eligible
