8

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: 

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

But other WordPress blogs have lots of tags included in it.

Also, my XML file's sitemap name looks like this: sitemap.xml.gz.

How do I add other possible parameters in my robots.txt file? Is the XML file sitemap.xml.gz name valid, and what is the difference between a .xml and .gz extension?

Improve this question
0

2 Answers 2

Reset to default
9

Yes, sitemap.xml.gzis a valid name. The .gz extension just means that it's been compressed (using gzip compression), so that it's smaller and served faster. Most search engine bots can read gzip'd compressed content.

You can edit the sitemap's XML content just by decompressing sitemap.xml.gz using a compression utility like 7-zip.

"Robots.txt" is separate from the sitemap.xml, and can be edited with a text editor (without decompressing it). There are many parameters that you can add to the "robots.txt" file, and depends on what you would like to achieve. For more information on possible parameters, Google "robots.txt", or check this link: Robots.txt

Improve this answer
1
1

XML is the markup language (Extensible Markup Language) and gz (short for GZip) is the compression format in this case. All sitemaps must be in xml format like sitemap.xml or in a compressed xml format like sitemap.xml.gz.

Also your sitemap's url should be the very first line in your robots.txt file and it should read:

Sitemap: http://www.example.com/sitemap.xml

or in your case:

Sitemap: http://www.example.com/sitemap.xml.gz

The purpose of using G-Zip is to increase your pagespeed which is great for SEO and increasing rank. On a side and similar note, your website should also have a readable page that is accessible to your users called

sitemap.html

if you're using html.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.