Questions tagged [sitemap]

A site map (or sitemap) is a list of pages of a web site accessible to crawlers or users.

Filter by
Sorted by
Tagged with
269 votes
20 answers
44k views

The Sitemap Paradox

We use a sitemap on Stack Overflow, but I have mixed feelings about it. Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow ...
Jeff Atwood's user avatar
  • 13.9k
3 votes
3 answers
2k views

HTTP to HTTPS: Wait for new sitemap to be indexed?

Running HTTP and HTTPS versions of the site side-by-side. HTTP is canonical right now. Goal is to change that to HTTPS. Created a new site in Search Console and submitted a new sitemap 4 days ago. ...
freehat12's user avatar
30 votes
4 answers
14k views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
Stephen Ostermiller's user avatar
13 votes
4 answers
26k views

Sitemaps - one per subdomain or one for the base domain?

I have more than 5 subdomains on a project, say developer, www, staff and some more, what is the best practice in creating sitemaps? All subdomains that share a same sitemap at base domain (example....
x0x's user avatar
  • 245
9 votes
4 answers
10k views

Google doesn't seem to update the description or title of my homepage

Before we launched our website, we had set up a "coming soon" page and google picked up the title and description from its contents. So the description in the search results said: Coming soon! ...
epicwhale's user avatar
  • 191
4 votes
1 answer
5k views

How to fix "404 Not Found" crawl errors reported by Google in Search Console

I was checking on Google webmaster tools for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
Freeme's user avatar
  • 41
7 votes
3 answers
3k views

Google says my sitemap index was "processed successfully", yet "total discovered URLs" is 0

I submitted a new version of the sitemap index for my site in Google Search Console two days ago. Soon after submission, GSC started saying that the index was "processed successfully", and yet has ...
Jan Żankowski's user avatar
5 votes
6 answers
1k views

Looking for a sitemap generator

I am looking for a good way to generate a sitemap for my current site. I inherited a huge site at work and we are going to start the process of IA analysis and redesign. It would be awesome if there ...
Purcell's user avatar
  • 270
5 votes
6 answers
6k views

How to tell Google that I have changed my website URLs?

I have done major updates to my website, and renamed all my URLs. How can I tell Google that I have renamed these URLs and let Google refresh its index? My website URL: http://www.pndmasr.com/ My ...
Zalaboza's user avatar
  • 153
2 votes
2 answers
2k views

Why the new Google Search Console does not read my sitemap?

I have the following problem: I cannot submit any sitemap recently to Google Search Console. Let me give you an example: https://ru.azrhymes.com/static/sitemaps/ru/sitemap-main-1.xml. (validator says ...
Barney Szabolcs's user avatar
11 votes
7 answers
10k views

Does Google penalize daily updated <lastmod> tags in sitemaps if the data is not daily updated?

I've got a sitemap that is generated daily with a lot of links to product pages. These products are imported daily from another data source. Because the update consists of throwing away all current ...
Elicit's user avatar
  • 173
7 votes
3 answers
543 views

Shall I submit sitemap containing all the bad (now 404) URLs after the spam URL hack?

Earlier this month, one of my sites got hacked and an attacker created near 1 million spam URLs. The site has been fixed relatively fast, however, after one month Google search results still shows ...
RhymeGuy's user avatar
  • 419
5 votes
1 answer
77 views

SEO consequences for category pages 404-ing

Question, is it ok for one of the pages levels to 404? (SEO-wise) Let's say I have a sitemap like this: example.com --/services --/foo --/bar --/sectors --/my-sector --/another-sector --/...
Ben Sterling's user avatar
3 votes
1 answer
121 views

Warning after submitting sitemap to Google: "We found that some URLs redirect to other locations"

I noticed that I have three warnings when I submitted my Sitemap. They all say this: When we tested a sample of URLs from your Sitemap, we found that some URLs redirect to other locations. We ...
Desi's user avatar
  • 31
73 votes
5 answers
30k views

Are there any clear indicators that my sitemap file is beneficial?

I have recently created a sitemap.xml file and uploaded it to my Google Webmasters Tools account. Google didn't report any issues or errors with the uploaded sitemap of my site. Now my question is: ...
user avatar
15 votes
1 answer
6k views

Can I host a sitemap on another domain?

Just out of curiosity, I have developed a small app analysing and generating sitemaps for a given domain. Can I tell search engines to look for the sitemap of a given domain on my app using a ...
iammichiel's user avatar
4 votes
1 answer
886 views

Should paginated sites be included in sitemap?

My website is ad listing for boats, boats' engines, boat rentals, etc (www.mynautics.com). There are ads listed on multiple pages. I have finished (not jet seen on link above) pagination: added prev ...
M.V.'s user avatar
  • 177
2 votes
1 answer
791 views

Sitemaps & <link rel=... for quad-directional websites

I realize there are many similar questions, but I think this consolidates multiple points of confusion. This question follows from the great advice given here. I have a site that has a separate mobile ...
asimovwasright's user avatar
1 vote
1 answer
1k views

Sitemap update lastmod and ping. what is the advantage?

My website has 6milions pages of companies. ex: www.mywebsite.com/company-a I will generate one sitemap index file and put the url in robots.txt When the company A update one information in your ...
oniram's user avatar
  • 13
1 vote
4 answers
907 views

How to remove indexed pages from google effectively

My site was getting traffic and it was on the rise, then I added https and somewhere in that time I also created tons of extra link for google to crawl, thinking this will increase traffic in those ...
TheTechGuy's user avatar
8 votes
3 answers
10k views

How to submit sitemap to Baidu?

I've some pages in my site in Chinese but I don't understand Chinese (it's some GIS data). Is there any way to submit sitemap to Baidu?
AgA's user avatar
  • 1,448
7 votes
1 answer
294 views

If a website is accessed on HTTPS instead of HTTP, should the URLs on the sitemap be HTTPS or HTTP?

I have changed my website from HTTP to HTTPS. At this moment, I have the next rewrite rule working to redirect calls for HTTP to HTTPS: RewriteEngine On RewriteCond %{SERVER_PORT} 80 RewriteRule ^(...
toofast's user avatar
  • 107
6 votes
4 answers
6k views

Google Webmaster Tools says my XML sitemap "appears to be an HTML page"

We're running a lot of sites and we've started to get a lot of these errors in Webmaster Tools: Sitemap is HTML Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead. ...
Adam Lynch's user avatar
5 votes
2 answers
1k views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question is ...
CP3O's user avatar
  • 679
4 votes
1 answer
3k views

Submitting a URL to Google using PHP/cUrl

I've been successfully submitting xml sitemaps to Google using cURL. The URL is http://www.google.com/webmasters/sitemaps/ping?sitemap= The problem is that XML sitemaps take several days to be ...
Robin LeBon's user avatar
4 votes
1 answer
4k views

Unsupported file format of sitemap - Google

I'm having an issue where I'm submitting my sitemap, too bad I keep this omitted error: Unsupported file format Your Sitemap does not appear to be in a supported format. Please ensure it meets our ...
MacMac's user avatar
  • 237
3 votes
3 answers
6k views

When should I update lastmod value in the sitemap?

I have a website like Stackoverflow. I'm creating a sitemap for it. I want to know, what timestamp should be as the value of last-modified for a page? (a question and all its answers) All I'm trying ...
Martin AJ's user avatar
  • 393
3 votes
1 answer
994 views

Does name of sitemap file that Wordpress generates matter?

The Yoast SEO plugin for wordpress creates an XML sitemap with the file name sitemap_index.xml. There is another sitemap plugin (Google XML Sitemaps) in the Wordpress site I'm working on that creates ...
Cave Johnson's user avatar
3 votes
2 answers
10k views

Free unlimited-page sitemap generator? [duplicate]

Possible Duplicate: Looking for a sitemap generator I have been using www.xml-sitemaps.com/ but their limit is 500 pages. My site has grown beyond that and I am wondering whether there is ...
Genadinik's user avatar
  • 1,264
3 votes
1 answer
111 views

How should I choose which URLs I want indexed to include in my sitemap?

I am working on a sitemap. I have read it should contain the URLs that you want indexed by Google. But which pages do I want indexed? It seems to me that makes sense to index pages such as the root ...
Bernardo Benini Fantin's user avatar
3 votes
1 answer
2k views

Should I change the URLs in the XML sitemap after 301 redirecting them?

I am using 301 redirects for all URLs. Should I change sitemap URLs now? The sitemap currently contains the following URLs: example.com/cat/1.html example.com/cat/2.html example.com/cat/3.html ...
Ario's user avatar
  • 143
1 vote
1 answer
215 views

Google Search Console Adding Sitemap "Couldn't fetch" and "Sitemap could not be read" txt file

I made a fairly simple .txt file I am getting the following errors I find it very odd. The 2.txt is just the .txt with fewer than 50,000 results. Somehow it seems like Google can read the longer .txt ...
David Kong's user avatar
1 vote
2 answers
910 views

Do Google sitemap URLs need to be double urlencoded?

Do Google sitemap URLs need to be double urlencoded? I am submitting my sitemaps to Google and Google is having difficulty navigating to URLs that contain urlencoded characters. An example of a ...
Hoytman's user avatar
  • 449
1 vote
1 answer
6k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
user avatar
0 votes
2 answers
533 views

Question about paginated based URLs and category URLs with XML sitemaps

I have a blog site that has this URL structure: http://website.com/category/ http://website.com/category/page1/ http://website.com/category/subcat/ http://website.com/category/article_title A flow ...
Dan's user avatar
  • 363
16 votes
4 answers
5k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
Cesar's user avatar
  • 405
15 votes
4 answers
9k views

How long does Bingbot typically take to process a sitemap submission?

I submitted a domain's sitemap.xml to Bingbot (in Webmaster Tools). It's been a couple of days but the status is still pending. Does anyone know how much time does Bing Webmaster Tools takes to ...
vulcan raven's user avatar
11 votes
1 answer
2k views

Internal links to pages vs. sitemap.xml links to pages

I have a page that is not linked to from anywhere, but it is listed in sitemap.xml. It will be crawled and display in the SERP. Would it be preferable, from a SEO perspective, to have the page ...
nathanziarek's user avatar
10 votes
2 answers
1k views

Can an XML sitemap file by generated dynamically, or does it have to be static?

Does it matter if I serve sitemap.xml as a dynamic page? e.g., use .htaccess to rewrite sitemap.xml (or whatever static URL I submitted at the Webmaster Tools) to a PHP file. The PHP file will call ...
Joel's user avatar
  • 673
9 votes
2 answers
4k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
SameOldNick's user avatar
8 votes
3 answers
2k views

Are there specific advantages or disadvantages of an XML sitemap over a TXT based sitemap?

Are there any specific advantages or disadvantages in generating an XML based sitemap in place of a simple txt based sitemap (list of URLs)? I realize that in the XML format I can set priority and ...
cboettig's user avatar
  • 325
8 votes
1 answer
1k views

Which pages from the sitemap are not indexed by Google?

I have a sitemap file that has been submitted to Google, and has been processed, according to Google Webmaster Tools. As per GWT for 2013-05-31, 8255 web-pages are submitted, but only 8209 are ...
cnst's user avatar
  • 972
7 votes
2 answers
2k views

Is it possible to make an XML sitemap pretty enough to show to users?

I wouldn't mind showing my XML sitemap to users if there were a way to make it human usable as opposed to just machine readable. Is it possible to: Choose the colors and layout Make the locations ...
Stephen Ostermiller's user avatar
7 votes
4 answers
230 views

Sitemaps HTML and others [closed]

I am looking for the best resources on how to build sitemaps. I am currently aware of the obvious HTML sitemaps and the (google) sitemap.xml and (yahoo) urllist.txt Anyone know of resources for ...
Lizard's user avatar
  • 281
7 votes
3 answers
1k views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
Oleg Shemetov's user avatar
7 votes
3 answers
6k views

Sitemap: Hierarchical layout possible?

Is it possible to design my sitemap hierarchically? My site is laid out hierarchically into World=>Continent=>Country=>State=>City=>Articles Can I design my sitemap xml like this?...
Quintin Par's user avatar
6 votes
3 answers
3k views

How long does it take for Google Webmasters to index site after submiting sitemap? [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have submitted my website's sitemap to webmaster's tools. The status on the sitemap says OK and it shows that 12 URLs ...
user avatar
6 votes
2 answers
4k views

Can a sitemap index file list sitemaps in a different directory?

If I have a sitemap index, can the sitemaps listed in the index be located in a subdirectory, but have links that isn't specific to the subdirectory? I'm trying to avoid having multiple sitemaps in ...
RHPT's user avatar
  • 203
6 votes
2 answers
1k views

What document types can be included in an XML sitemap?

AFAIK, Google accepts sitemaps which are about ordinary URLs (text/plain), images (image/png, image/gif, image/jpg, etc.), and videos (video/avi, video/mpeg, etc.). My question is, how many Internet ...
Saeed Neamati's user avatar
6 votes
1 answer
2k views

Flat vs. Hierarchical Tree Structure

I am a developer in the middle of the development of a medium-sized website. A requirement that we just received from the resident SEO person on the project is for all the URLs on the site to be just ...
raynjamin's user avatar
  • 161