New answers tagged web-crawlers
If screaming frog is finding the sub-domain, there must still be a link to it on one or more of your pages. That's how it discoverers pages, following links that are in the source code of the page. So if the other 2 sub-domains aren't coming up it means they most likely aren't linked on the site. (although if they are blocked in robots.txt and you do not ...
What I don't know is whether it also blocks the URL from showing up in search results, or just that particular content with that URL. If you have a noindex robots tag then the URL should not appear in the search results. (It should be noted, however, that if you block the crawling of this URL, eg. with robots.txt, then Google won't be able to see the ...
In the Google database, the single most important table of the schema is the document table which hosts the URL and document id. All other database tables and data elements rely completely on this. While there is a unique document id (allows for a smaller index size), the URL uniquely identifies a page since there cannot be two pages on any given URL. ...
You need to add the every unique URL to the XML Sitemaps.This helps in Google bots to crawl your websites in depth. And also add different types of Sitemaps https://support.google.com/webmasters/answer/156184?hl=en which is associated with every product.
You do not want your XML sitemap to include duplicate content. In your case, it sounds like that one product exists at multiple URLs, which is the very definition of duplicate content. So, what I'd say is that you want to have the page mywebsite.com/product1 (the canonical version of the URL) in your XML sitemap, then keep the others out of your XML sitemap. ...
If you haven't already done so, register an account with Google Webmaster Tools and add your domain to it then access your domain, and select the gear icon and visit Site Settings then you can select "Limit Google's maximum crawl rate" and select a value that you're OK with. Probably the lowest value works well for you which may be 0.002 requests a second (a ...
This is not a good setup. Canonical pointing to page with different canonical will surely lead to problems (including indexation and rankings), sooner or later. The best scenario in your particular example is to 301 redirect the http://www.example.com/phone/great-page.html to http://www.example.com/phone/even-greater-page.html. The best practice is to use ...
Anything in an iframe can't be indexed in the first place. So if you're not wanting the page to be found by what is said in the reveiws, an iframe is definitely the way to go as it will also make it so that you won't have to worry about duplicate content.
Top 50 recent answers are included