New answers tagged

0

Allow: * should actually be be Allow: /. That could be the issue... because, really, using Allow: is somewhat meaningless by itself -- its purpose is for allowing a sub-path within a denied path. As it stands, it seems possible that your file is being misinterpreted. Anything not denied is supposed to be implicitly allowed. If you want to allow ...


0

You can't ask crawler the frequency with which it should crawl your page. Though you can define the change frequency in your sitemap but that too does not ensure any defined behavior. Suppose your sitemap had 100 links and not it has 200 links. The crawling of delta of these links depends on the actual content change on your pages, the domain authority of ...


2

Most crawlers tend to ignore the lastmod tag as many webmasters do a poor job of keeping it up-to-date and forget to update it when they update content on the site. A sitemap is still important to identify content on the site but even if the lastmod tag shows the content hasn't been updated since the last crawl the crawler will still crawl it to confirm the ...


1

You have to read this question carefully. It is about the Google Custom Search Engine which is not the Google Search Engine. It is a feature whereby a site can have a custom search engine for the site. This, at least, is a site search feature. These are two different things though there is some relationship between the two. Assuming that a site owner opts ...


1

If you want your pages indexed faster you need to do more than add them to your sitemap. A sitemap doesn't pass any link juice to the pages and Google usually doesn't index pages that have no link juice. See The Sitemap Paradox for more information. You need to link new pages from other pages on your site. It might be worth having a "recently ...


2

You can easily achieve this using the same doc_root without needing to worry about uploading the sitemap to the S3 bucket. Using the following rewrite code RewriteEngine On RewriteCond %{HTTP_HOST} ^(domain1\.com)$ RewriteRule ^sitemap\.xml$ /domain1-sitemap.xml [L] RewriteCond %{HTTP_HOST} ^(domain2\.com)$ RewriteRule ^sitemap\.xml$ /domain2-sitemap.xml ...


3

There is no fixed time from when Google crawls a site to when it is available on the index. A few years ago it would have taken anywhere up to 6 weeks for content to be added due to Google rebuilding the main index in batches, now with the live updating it is meant to take longer but it could be added to the index any time from a few days to a few weeks ...


1

Unfortunately there is no way to submit a single URL using cURL or anything automated such as what you are suggesting and there is no way to speed up the indexing process for either the content or the images. Neither sitemap submission nor manual URL submission guarantee the content will be added to the index. With the manual fetch it can update the index a ...


1

I simply compressed them using gzip: gzip sitemap1.xml My server nginx is enabled to send compressed files. It looks like you're trying to send a double-compressed file and a compressed file. When you gzip a file and host it on a server that is set up to compress and send files, then you basically compressed the same file twice and when one ...


0

If you try to download your own sitemap files, e.g. example.xml.gz, most web browsers will correctly prompt you to download the compressed sitemap file. However, if your webserver is not sending correct MIME headers for .gz files, some browsers may instead try open the sitemap file. This will fail as the internet browser has not correctly recognized the ...



Top 50 recent answers are included