Hot answers tagged

10

You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules. RewriteRule ^robots\.txt$ /robots.php [L] That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...


5

Yes, the same way any request can be "dynamic". However, you would not redirect (as in your example code), you should internally rewrite using mod_rewrite. (The same as what Drupal is probably already doing.) For example, in your root .htaccess file: RewriteEngine On RewriteRule ^robots\.txt$ robots.php [L] RewriteEngine should only occur once (although ...


5

You generally won't want these excess pages in your sitemap. The sitemap should only contain links to URLs you actually indexed & listed by search engines. If you have property/form URLs in your sitemap, you've probably used a 3rd party Sitemap Generator. These just crawl all links on your website indescriminately, whereas something like ...


5

Can I list sitemap.php in the robots.txt? No, that wouldn't work. The XML Sitemap goes in robots.txt. The public-facing site map will be crawled by search engines like any other page.


4

You're better off creating a multilingual sitemap, just to avoid any source of confusion. The format you have shown is correct. In fact, you could even drop the hreflang declarations in your page sections and just use the declaration within the sitemap. The Official Webmaster Tools blog explains the advantages of using the multilingual sitemaps ...


3

There are no SEO benefits to serving a compressed XML sitemap over serving one that is not compressed. The advantage of compression is simply to save bandwidth and the time it takes to download. (If your sitemap is huge.) Note that the limits for the size of the sitemap are the uncompressed size (ie. 50MB uncompressed for Google).


3

This post is 4 years old, hope the status in not Pending anymore. But this helped me. After 14 days of pending status, I found this link, to PING bing with your sitemap. It's was a kind of awake call for my sitemap. Few hours later, my site was indexed. Upload yous sitemap (but you already did hence the pending status) Typ in your browser: ...


3

This is not an answer, but a caveat for XML sitemap wizard users. I recently did a tech & SEO overview of a site rebuild where an XML sitemap was auto-generated by a wizard. She failed to review it and see that all the URL's were like **new.**site.com/page.asp - the dev server where the site was built! Typing out a txt sitemap has the advantage of ...


3

It seems like you have your HTTP and HTTPS sites redirect to the HTTPS version of your www.example.com. This is good. Just know there is a cost for this initially. In your case, only one sitemap will ever be seen. It is the HTTPS version of your www.example.com site. Any sitemap should reflect both HTTPS and www in the URLs that are listed. Any other URL ...


3

You have all the steps correct, but it won't prevent a temporary (up to 8 month) drop in SEO traffic. Google does not support migrating an entire folder to its own new domain. When you do so, your new domain name will enter the "sandbox" for a new domain and Google won't trust it as much as the content in the folder on your main domain for some months. ...


3

There is no fixed time from when Google crawls a site to when it is available on the index. A few years ago it would have taken anywhere up to 6 weeks for content to be added due to Google rebuilding the main index in batches, now with the live updating it is meant to take longer but it could be added to the index any time from a few days to a few weeks ...


3

First off, you are dealing with a few things in your question. I will try and go through it for you in no particular order. As far as sitemaps go, they have nothing to do with a pages or sites performance generally. It is just a way to tell Google what pages you have. Where sitemaps are necessary is when part of your site cannot be crawled by a search ...


2

Please understand that Google will largely ignore sitemaps if a site can be effectively crawled. Google will compare what it can crawl with the sitemap to ensure that it can properly crawl the site. A sitemap is only necessary if a site is extremely large, the content cannot effectively be linked, or if content resides behind a paywall or login. If you ...


2

Google has a help document about sitemaps in which they list the various types: web video image mobile news


2

The problem is that Google hates pages in the sitemap that do any kind of redirect. So, try something from this few options that crossed my mind: Don't let the page in the sitemap to redirect (the pages must give status code 200 - OK) Change the pages in the sitemap with the landing pages. Exclude redirect pages in sitemap.


2

having a static image's url / path would help your image to appear in image search results. without having an image url it can't be cached properly. yes, i would strongly recommend to use images like images, making use of real image files and implement them with img src alt + title or css.


2

Create a sitemap containing URLs for the non-www version of the site that you want indexed. Anyone who accesses the www version should get redirected to the non-www version of the same URL. If your server is apache with mod_rewrite installed, you can modify your server configuration so it also contains the following lines: RewriteEngine On RewriteCond ...


2

A Google search led me to this program: XML comparison tool It's free and online, let us know how you get on.


2

You can't force Google to crawl/rescan your sitemap instantly. Once Google index any of webpage from your website, it start crawling again and again i.e. they crawl webpages from their own indexed database. So if you removed those pages from sitemap, even from your website which you linked to, then still that pages will able to crawl for Googlebot. ...


2

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to identify content on the site but even if the lastmod tag shows the content hasn't been updated since the last crawl the crawler will still crawl it to confirm the ...


2

No you do not need to keep them in the root folder, they do not need a file extension either. You should however have a redirect in place so that yourdomain.com/sitemap.xml points to wherever it's available at. This ensures that bots besides those you tell are able to find it via that common path. Here are a few examples I can think of that work for ...


2

Google Sitemap Compression Support It's interesting to read Google's decision to use poorly worded documentation on what sitemaps are or are not supported. Here is a quote from Google Webmaster Tools regarding an error message received when uploading a archived file. Google couldn't process your sitemap because we couldn't uncompress it. This error ...


2

As Closetnoc mentioned in his comment, GZ is just a compressed version of your text sitemap. So my answer to your question Should I include just the .gz file in the robots.txt or both? There is no point in submitting both sitemap.xml as well as sitemap.xml.gz. Most of the search engine bots can read the compressed content. So if you are submitting ...


2

I originally read you question as the number of indexed pages from your sitemap is 1500 and that you are worried. Then I thought about it. Perhaps there are things you need to know. If your site is new, then it will take quite a while for a new site to be indexed. I am not sure how old your site is, but it can take as much as a year for some sites to sink ...


2

You do not want your XML sitemap to include duplicate content. In your case, it sounds like that one product exists at multiple URLs, which is the very definition of duplicate content. So, what I'd say is that you want to have the page mywebsite.com/product1 (the canonical version of the URL) in your XML sitemap, then keep the others out of your XML sitemap. ...


2

Making the sitemap file dynamic is fine -- it's a good way to auto-update your sitemaps. Making the robots.txt file dynamic (for the same host! Doing this for separate hosts is essentially just a normal robots.txt file for each of them.) would likely cause problems: it's not crawled every time a URL is crawled from the site, so it can happen that the ...


2

You can easily achieve this using the same doc_root without needing to worry about uploading the sitemap to the S3 bucket. Using the following rewrite code RewriteEngine On RewriteCond %{HTTP_HOST} ^(domain1\.com)$ RewriteRule ^sitemap\.xml$ /domain1-sitemap.xml [L] RewriteCond %{HTTP_HOST} ^(domain2\.com)$ RewriteRule ^sitemap\.xml$ /domain2-sitemap.xml ...


1

you can use alt in image sitemap, but only as <image:caption>, as mentioned in Google's documentation


1

The solution you're looking for is actually a lot less painstaking than your example makes it seem. I'll start from the top and work my way down with explanations. For sake of ease, I'm going to assume the website is constructed from static files hosted on a remote fileserver. Point #1 - Structure: I would strongly encourage a bit of restructuring. Create ...


1

You can use Google image extensions for sitemaps to give Google more information about the images available on your pages. Image sitemap information helps Google discover images that we might not otherwise find (such as images your site reaches with JavaScript code), and allows you to indicate images on your site that you want Google to crawl and index. You ...



Only top voted, non community-wiki answers of a minimum length are eligible