Hot answers tagged

10

You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules. RewriteRule ^robots\.txt$ /robots.php [L] That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...


5

You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines. If you have property/form URLs in your sitemap, you've probably used a 3rd party Sitemap Generator. These just crawl all links on your website indescriminately, whereas something like https://www....


5

Yes, the same way any request can be "dynamic". However, you would not redirect (as in your example code), you should internally rewrite using mod_rewrite. (The same as what Drupal is probably already doing.) For example, in your root .htaccess file: RewriteEngine On RewriteRule ^robots\.txt$ robots.php [L] RewriteEngine should only occur once (although ...


5

Can I list sitemap.php in the robots.txt? No, that wouldn't work. The XML Sitemap goes in robots.txt. The public-facing site map will be crawled by search engines like any other page.


4

You're better off creating a multilingual sitemap, just to avoid any source of confusion. The format you have shown is correct. In fact, you could even drop the hreflang declarations in your page sections and just use the declaration within the sitemap. The Official Webmaster Tools blog explains the advantages of using the multilingual sitemaps ...


4

A sitemap is so that Google can index your pages without having to scrape and investigate every internal link. If you want something indexed, it should be in there. If you don't want it indexed - robots.txt it. It doesn't matter if you use multiple sitemaps or a single large file. There's no limits, advantages, disadvantages outside of management.


3

This is not an answer, but a caveat for XML sitemap wizard users. I recently did a tech & SEO overview of a site rebuild where an XML sitemap was auto-generated by a wizard. She failed to review it and see that all the URL's were like **new.**site.com/page.asp - the dev server where the site was built! Typing out a txt sitemap has the advantage of ...


3

This post is 4 years old, hope the status in not Pending anymore. But this helped me. After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call for my sitemap. Few hours later, my site was indexed. Upload yous sitemap (but you already did hence the pending status) Typ in your browser: http://www....


3

There are no SEO benefits to serving a compressed XML sitemap over serving one that is not compressed. The advantage of compression is simply to save bandwidth and the time it takes to download. (If your sitemap is huge.) Note that the limits for the size of the sitemap are the uncompressed size (ie. 50MB uncompressed for Google).


3

First off, you are dealing with a few things in your question. I will try and go through it for you in no particular order. As far as sitemaps go, they have nothing to do with a pages or sites performance generally. It is just a way to tell Google what pages you have. Where sitemaps are necessary is when part of your site cannot be crawled by a search ...


3

You have all the steps correct, but it won't prevent a temporary (up to 8 month) drop in SEO traffic. Google does not support migrating an entire folder to its own new domain. When you do so, your new domain name will enter the "sandbox" for a new domain and Google won't trust it as much as the content in the folder on your main domain for some months. ...


3

There is no fixed time from when Google crawls a site to when it is available on the index. A few years ago it would have taken anywhere up to 6 weeks for content to be added due to Google rebuilding the main index in batches, now with the live updating it is meant to take longer but it could be added to the index any time from a few days to a few weeks ...


2

As Closetnoc mentioned in his comment, GZ is just a compressed version of your text sitemap. So my answer to your question Should I include just the .gz file in the robots.txt or both? There is no point in submitting both sitemap.xml as well as sitemap.xml.gz. Most of the search engine bots can read the compressed content. So if you are submitting ...


2

I originally read you question as the number of indexed pages from your sitemap is 1500 and that you are worried. Then I thought about it. Perhaps there are things you need to know. If your site is new, then it will take quite a while for a new site to be indexed. I am not sure how old your site is, but it can take as much as a year for some sites to sink ...


2

Making the sitemap file dynamic is fine -- it's a good way to auto-update your sitemaps. Making the robots.txt file dynamic (for the same host! Doing this for separate hosts is essentially just a normal robots.txt file for each of them.) would likely cause problems: it's not crawled every time a URL is crawled from the site, so it can happen that the "...


2

You do not want your XML sitemap to include duplicate content. In your case, it sounds like that one product exists at multiple URLs, which is the very definition of duplicate content. So, what I'd say is that you want to have the page mywebsite.com/product1 (the canonical version of the URL) in your XML sitemap, then keep the others out of your XML sitemap. ...


2

The problem is that Google hates pages in the sitemap that do any kind of redirect. So, try something from this few options that crossed my mind: Don't let the page in the sitemap to redirect (the pages must give status code 200 - OK) Change the pages in the sitemap with the landing pages. Exclude redirect pages in sitemap.


2

Please understand that Google will largely ignore sitemaps if a site can be effectively crawled. Google will compare what it can crawl with the sitemap to ensure that it can properly crawl the site. A sitemap is only necessary if a site is extremely large, the content cannot effectively be linked, or if content resides behind a paywall or login. If you ...


2

Google has a help document about sitemaps in which they list the various types: web video image mobile news


2

having a static image's url / path would help your image to appear in image search results. without having an image url it can't be cached properly. yes, i would strongly recommend to use images like images, making use of real image files and implement them with img src alt + title or css.


2

Create a sitemap containing URLs for the non-www version of the site that you want indexed. Anyone who accesses the www version should get redirected to the non-www version of the same URL. If your server is apache with mod_rewrite installed, you can modify your server configuration so it also contains the following lines: RewriteEngine On RewriteCond %{...


2

No you do not need to keep them in the root folder, they do not need a file extension either. You should however have a redirect in place so that yourdomain.com/sitemap.xml points to wherever it's available at. This ensures that bots besides those you tell are able to find it via that common path. Here are a few examples I can think of that work for sitemaps:...


2

A Google search led me to this program: XML comparison tool It's free and online, let us know how you get on.


2

You can't force Google to crawl/rescan your sitemap instantly. Once Google index any of webpage from your website, it start crawling again and again i.e. they crawl webpages from their own indexed database. So if you removed those pages from sitemap, even from your website which you linked to, then still that pages will able to crawl for Googlebot. ...


2

You can easily achieve this using the same doc_root without needing to worry about uploading the sitemap to the S3 bucket. Using the following rewrite code RewriteEngine On RewriteCond %{HTTP_HOST} ^(domain1\.com)$ RewriteRule ^sitemap\.xml$ /domain1-sitemap.xml [L] RewriteCond %{HTTP_HOST} ^(domain2\.com)$ RewriteRule ^sitemap\.xml$ /domain2-sitemap.xml [...


2

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to identify content on the site but even if the lastmod tag shows the content hasn't been updated since the last crawl the crawler will still crawl it to confirm the ...


2

In your situation, Don't include "www" because web crawlers that do read the sitemap directive from your robots.txt will attempt to access the "www" version of your sitemap and then be redirected to the non-www version of the URL. Not only will you be wasting a couple extra bytes by including "www", but the crawlers that parse the sitemap directive ...


2

So long as the entries in the sitemap point to the correct URLs and Google knows to access the sitemap file and is doing so, you're fine. The name of that file is not relevant to the SEO. In other words, my sitemap could be www.mysite.com/f*ckyougoogle.xml and neither the spider nor the algorithm would care.


2

Add 301 redirects from your old URLs to your new ones. (and google's index will update itself)


2

I'd suggest updating the links in your website to the absolute URL and avoid redirections (unless of course you have valuable links pointing to the URL ending with the ID) This is an unnecessary redirection and should better be avoided.



Only top voted, non community-wiki answers of a minimum length are eligible