I know in Google Webmaster Tools, under "search appearance" for any domain, there's an "HTML Improvements" section which only checks crawled URLs specifically for that domain. My website has subdomains. Is there a way I can find out whether two titles or meta descriptions are duplicate between any two URLs where the subdomain each URL belongs to is different? The only thing I can think of at this point is to programmatically access each and every url from every subdomain which takes tedious amounts of time since I have over a total of about 3 million URLs. On top of that, It will take bandwidth and/or server processing power which can be better spent for actual users.

  • 1
    Not possible, since data in Webmaster tools is not shared a cross domains, each domain and sub domain is treated as a new site. If you want to find duplicate titles and descriptions, then you need to use a 3rd party app. Also, if the content is unique on each page then it doesn't matter if titles are duplicate. Titles are duplicated in the thousands daily due to the fact the title tag is only so many characters and duplicates regularly occur naturally, this isn't a factor. Oct 7, 2015 at 20:05
  • 1
    Also, if you do decide to use a crawler then I recommend that you copy the database locally, 3 million isn't a lot of pages when its local. Then you simply keep your local copy up to date using increment updates. Oct 7, 2015 at 20:18
  • @SimonHayter has a good point making a local copy. I do that on my Netbook and use something like ScreamingFrog or SEO PowerSuite also on the Netbook for audits. It will take a while of course. However, it is easy enough to set-up and go to bed and review the results in the morning. If my Netbook can handle it as easily as it does, any PC should with without breaking a sweat. But keep a towel handy just in case... ;-)
    – closetnoc
    Oct 7, 2015 at 21:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.