Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sitemap]

A site map (or sitemap) is a list of pages of a web site accessible to crawlers or users.

245
votes
19answers
35k views

The Sitemap Paradox

We use a sitemap on Stack Overflow, but I have mixed feelings about it. Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow ...
3
votes
1answer
5k views

How to fix “404 Not Found” crawl errors reported by Google in Search Console

I was checking on Google webmaster tools for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
8
votes
4answers
6k views

Google doesn't seem to update the description or title of my homepage

Before we launched our website, we had set up a "coming soon" page and google picked up the title and description from its contents. So the description in the search results said, "Coming soon! Visit ...
8
votes
4answers
12k views

Sitemaps - one per subdomain or one for the base domain?

I have more than 5 subdomains on a project, say developer, www, staff and some more, what is the best practice in creating sitemaps? All subdomains that share a same sitemap at base domain (example....
5
votes
6answers
692 views

Looking for a sitemap generator

I am looking for a good way to generate a sitemap for my current site. I inherited a huge site at work and we are going to start the process of IA analysis and redesign. It would be awesome if there ...
23
votes
4answers
9k views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
5
votes
6answers
4k views

How to tell Google that I have changed my website URLs?

I have done major updates to my website, and renamed all my URLs. How can I tell Google that I have renamed these URLs and let Google refresh its index? My website URL: http://www.pndmasr.com/ My ...
9
votes
7answers
8k views

Does Google penalize daily updated <lastmod> tags in sitemaps if the data is not daily updated?

I've got a sitemap that is generated daily with a lot of links to product pages. These products are imported daily from another data source. Because the update consists of throwing away all current ...
4
votes
3answers
1k views

HTTP to HTTPS: Wait for new sitemap to be indexed?

Running HTTP and HTTPS versions of the site side-by-side. HTTP is canonical right now. Goal is to change that to HTTPS. Created a new site in Search Console and submitted a new sitemap 4 days ago. ...
4
votes
1answer
45 views

SEO consequences for category pages 404-ing

Question, is it ok for one of the pages levels to 404? (SEO-wise) Let's say I have a sitemap like this: example.com --/services --/foo --/bar --/sectors --/my-sector --/another-sector --/...
72
votes
5answers
30k views

Are there any clear indicators that my sitemap file is beneficial?

I have recently created a sitemap.xml file and uploaded it to my Google Webmasters Tools account. Google didn't report any issues or errors with the uploaded sitemap of my site. Now my question is: ...
3
votes
1answer
552 views

Should paginated sites be included in sitemap?

My website is ad listing for boats, boats' engines, boat rentals, etc (www.mynautics.com). There are ads listed on multiple pages. I have finished (not jet seen on link above) pagination: added prev ...
9
votes
1answer
3k views

Can I host a sitemap on another domain?

Just out of curiosity, I have developed a small app analysing and generating sitemaps for a given domain. Can I tell search engines to look for the sitemap of a given domain on my app using a ...
7
votes
3answers
7k views

How to submit sitemap to Baidu?

I've some pages in my site in Chinese but I don't understand Chinese (it's some GIS data). Is there any way to submit sitemap to Baidu?
4
votes
1answer
2k views

Is it necessary to re-submit modified sitemaps?

Assuming a sitemap.xml has been registered properly to major players, do these players come and revisit those sitemap.xml from time to time to check for updates and modifications (say new entries are ...
6
votes
4answers
6k views

Google Webmaster Tools says my XML sitemap “appears to be an HTML page”

We're running a lot of sites and we've started to get a lot of these errors in Webmaster Tools: Sitemap is HTML Your Sitemap appears to be an HTML page. Please use a supported sitemap format ...
4
votes
1answer
480 views

Does name of sitemap file that Wordpress generates matter?

The Yoast SEO plugin for wordpress creates an XML sitemap with the file name sitemap_index.xml. There is another sitemap plugin (Google XML Sitemaps) in the Wordpress site I'm working on that creates ...
3
votes
2answers
5k views

Free unlimited-page sitemap generator? [duplicate]

Possible Duplicate: Looking for a sitemap generator I have been using www.xml-sitemaps.com/ but their limit is 500 pages. My site has grown beyond that and I am wondering whether there is ...
3
votes
1answer
1k views

Should I change the URLs in the XML sitemap after 301 redirecting them?

I am using 301 redirects for all URLs. Should I change sitemap URLs now? The sitemap currently contains the following URLs: example.com/cat/1.html example.com/cat/2.html example.com/cat/3.html ...
1
vote
2answers
512 views

Sitemap indexed count and Google results difference

My sitemap claims they have indexed over 3 thousand pages from it but when i do a site command in Google site:http://domain.com It returns 200 pages. I have heard of the sitemap index count ...
1
vote
2answers
708 views

Do Google sitemap URLs need to be double urlencoded?

Do Google sitemap URLs need to be double urlencoded? I am submitting my sitemaps to Google and Google is having difficulty navigating to URLs that contain urlencoded characters. An example of a ...
1
vote
1answer
501 views

Sitemap update lastmod and ping. what is the advantage?

My website has 6milions pages of companies. ex: www.mywebsite.com/company-a I will generate one sitemap index file and put the url in robots.txt When the company A update one information in your ...
1
vote
1answer
5k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
0
votes
2answers
478 views

Question about paginated based URLs and category URLs with XML sitemaps

I have a blog site that has this URL structure: http://website.com/category/ http://website.com/category/page1/ http://website.com/category/subcat/ http://website.com/category/article_title A flow ...
7
votes
4answers
213 views

Sitemaps HTML and others [closed]

I am looking for the best resources on how to build sitemaps. I am currently aware of the obvious HTML sitemaps and the (google) sitemap.xml and (yahoo) urllist.txt Anyone know of resources for ...
7
votes
2answers
1k views

Is it possible to make an XML sitemap pretty enough to show to users?

I wouldn't mind showing my XML sitemap to users if there were a way to make it human usable as opposed to just machine readable. Is it possible to: Choose the colors and layout Make the locations ...
4
votes
2answers
3k views

Can a sitemap index file list sitemaps in a different directory?

If I have a sitemap index, can the sitemaps listed in the index be located in a subdirectory, but have links that isn't specific to the subdirectory? I'm trying to avoid having multiple sitemaps in ...
14
votes
4answers
8k views

How long does Bingbot typically take to process a sitemap submission?

I submitted a domain's sitemap.xml to Bingbot (in Webmaster Tools). It's been a couple of days but the status is still pending. Does anyone know how much time does Bing Webmaster Tools takes to ...
10
votes
2answers
717 views

Can an XML sitemap file by generated dynamically, or does it have to be static?

Does it matter if I serve sitemap.xml as a dynamic page? e.g., use .htaccess to rewrite sitemap.xml (or whatever static URL I submitted at the Webmaster Tools) to a PHP file. The PHP file will call ...
7
votes
3answers
1k views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
6
votes
1answer
2k views

Flat vs. Hierarchical Tree Structure

I am a developer in the middle of the development of a medium-sized website. A requirement that we just received from the resident SEO person on the project is for all the URLs on the site to be just ...
5
votes
3answers
8k views

Sitemap to display hash tag links or not?

I am just creating sitemap for my site and wanted to find out if I should include # links for SEO? For Example: www.mydomain.com/#comment1 www.mydomain.com/#comment2 Does this help with SEO or ...
4
votes
1answer
6k views

How to add sitemap to my Blogspot blog? [closed]

I added a lot of pages in my blog. I need to track the blog traffic. If anyone know about adding sitemap to my blog, please send step by step tutorial for this process.
4
votes
2answers
3k views

Should 404 pages feature in your sitemap.xml

I tend to generate the sitemaps for my static sites (no CMS) by using a service such as xml-sitemaps.com. They way this works if you input your homepage and it crawls all the internal links of a site ...
4
votes
1answer
495 views

Determining which sitemap entries not indexed by google

We submit a sitemap to google and can see this being indexed in webmaster tools. We have 5140 entries, and have broken this up to 10 child site maps of 500 each. This all seems to be working well. ...
4
votes
2answers
647 views

sitemap that contains millions of links/URLs

I am currently working on a shopping / auction website. The involving party who's doing SEO consultancy is insisting to generate URLs for every single combination of products listed on the website, ...
4
votes
1answer
118 views

Moving folder's content to a new domain, what SEO steps to take?

So, we have a functionality on example.com/folder that we decided to make as separate site and we're in need of moving it to a new domain (brand it)... This is what I'm going to make SEO wise (...
3
votes
2answers
2k views

Google Search Console only showing one page in index status, while many pages indexed in sitemap

I am using Google webmaster tools for my site. I recently have changed address. Now, I wonder why the number of indexed pages in Crawl -> Sitemaps and Google Index -> Index Status are different?
3
votes
2answers
243 views

How should you inform search engines of the location of an XML sitemap?

There are 2 ways to make search engines aware of a sitemap's location that I know of: Include an absolute link to it in robots.txt Submit it to them directly. The relevant URLs are: http://www....
3
votes
3answers
1k views

Where can I find a list of search engine crawler user agents and their domain names? [closed]

I'm trying to follow stackoverflow's approach of using a white list for the sitemap. I have tried to find an extensive list of search engine crawlers (and other crawlers) to use in my white list but ...
3
votes
1answer
366 views

How should I approach sitemap.xml, hreflang and regions for my website

I am a developer on a large site which regionally targets some content with a specific url. Is it technically correct to target the countries in those regions in the sitemap.xml and point them to the ...
2
votes
1answer
98 views

Correctly handling multi language URL's in sitemaps

I have an English page: https://www.example.com/content/article-1 This page is also avaliable in other languages at the URLs: https://www.example.com/de/content/article-1 https://www.example.com/zh-...
2
votes
1answer
444 views

hreflang tags in sitemaps - your site has no hreflang tags

We are using hreflang tags in our sitemaps and we're currently experiencing a number of issues with errors in WMT suggesting 'no return tags' and 'your site has no hreflang tags' that don't seem ...
2
votes
2answers
1k views

Should we include images in sitemap to index?

The sitemap of my website is currently indexing only posts and pages. It is not indexing the media content that is images and attachments. Is it a good idea not to submit images for indexing. If not ...
1
vote
1answer
267 views

Sitemaps & <link rel=… for quad-directional websites

I realize there are many similar questions, but I think this consolidates multiple points of confusion. This question follows from the great advice given here. I have a site that has a separate mobile ...
10
votes
3answers
3k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
9
votes
2answers
4k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
8
votes
1answer
1k views

Which pages from the sitemap are not indexed by Google?

I have a sitemap file that has been submitted to Google, and has been processed, according to Google Webmaster Tools. As per GWT for 2013-05-31, 8255 web-pages are submitted, but only 8209 are ...
7
votes
2answers
1k views

Are there specific advantages or disadvantages of an XML sitemap over a TXT based sitemap?

Are there any specific advantages or disadvantages in generating an XML based sitemap in place of a simple txt based sitemap (list of URLs)? I realize that in the XML format I can set priority and ...
6
votes
1answer
265 views

If a website is accessed on HTTPS instead of HTTP, should the URLs on the sitemap be HTTPS or HTTP?

I have changed my website from HTTP to HTTPS. At this moment, I have the next rewrite rule working to redirect calls for HTTP to HTTPS: RewriteEngine On RewriteCond %{SERVER_PORT} 80 RewriteRule ^(...