New answers tagged

2

If the content never existed in the first place, you should return 404. Crawlers will know that there's nothing to see on that page and move on. If the content used to exist at that URL but no longer does (for example, a product was removed from the website), you should return 410. 404 means there is nothing at this URL. 410 means that there is no longer ...


2

First, list out the URLs to search in a text file. Use a utility like wget or aria to download the source codes to your local machine. Both these utilities have options to specify a list of URLs to download via text file as input. For example, wget uses --input-file=file option. Once downloaded to a folder on the local machine, you should be able to use a ...


1

I don't know of any search engine that would index your website's code. However, all is not necessarily lost. You could use a site-ripper to download the whole website to your computer, then use a tool capable of searching through text files for instances of a string you specify (free examples of both can be found online for your preferred OS). This would ...


4

You need two things: Time External links Googlebot will eventually increase its crawl rate and crawl your entire site. Getting 2 million pages crawled could take a year or two. Even then Google may choose to index a small percentage of them. Getting 90% of your pages indexed could take five years. The best way to speed up Google's crawl rate is by ...


1

Search engines don't index the same content on two different URLs regardless of whether or not you use canonical tags. Crawlers use the shingle algorithm to compare pages and can see when any two pages on the internet have substantially the same content. When search engines encounter duplicate pages they usually choose to index one of them and ignore the ...


1

The meaning of a canonical is to tell google which one is the original. Google will prefer to index the original one, and not the duplicate. Keep in mind: insert <link rel="canonical" href="example.com/a"/> also on the original page example.com/a (self reference) do not mix robots="noindex/follow" with the rel="...


2

inhales While X-Robots-Tag and meta robots are equivalent, robots.txt is different. The former is about indexing, while the latter is about crawling/visiting. Tell bots not to visit a URL by using robots.txt. Use only one of the three for each URL. Using both X-Robots-Tag and meta robots on a URL is redundant because they are equivalent, and using both ...


2

I would not recommend trying to set up your site to disallow everything except certain directories. The home page of your site will be blocked from crawling. Because most sites get many links to their home page, your site will be throwing away SEO value from a lot of incoming links. You will need to have direct external links to your deep crawlable ...


2

What you have would seem to be "about" correct, assuming you have the appropriate User-agent directive that precedes this? Disallow: / Disallow: /* However, you don't need to repeat the same directive, one with a trailing * and one without. robots.txt is prefix matching. The trailing * is superfluous and these match the same URLs. User-agent: * ...


1

You can get to Crawl Stats by using the nav bar to the left. It's under the "Legacy tools and reports" section, if that's what you're referring to. As far as manually indexing and crawling on a page-by-page basis, you can no longer access the old style of Google Search Console for those, you have to use the new version. For more information on how ...


Top 50 recent answers are included