Tag Info

Hot answers tagged

2

If screaming frog is finding the sub-domain, there must still be a link to it on one or more of your pages. That's how it discoverers pages, following links that are in the source code of the page. So if the other 2 sub-domains aren't coming up it means they most likely aren't linked on the site. (although if they are blocked in robots.txt and you do not ...


2

What I don't know is whether it also blocks the URL from showing up in search results, or just that particular content with that URL. If you have a noindex robots tag then the URL should not appear in the search results. (It should be noted, however, that if you block the crawling of this URL, eg. with robots.txt, then Google won't be able to see the ...


1

In the Google database, the single most important table of the schema is the document table which hosts the URL and document id. All other database tables and data elements rely completely on this. While there is a unique document id (allows for a smaller index size), the URL uniquely identifies a page since there cannot be two pages on any given URL. ...



Only top voted, non community-wiki answers of a minimum length are eligible