Skip to main content
added 251 characters in body
Source Link
Stephen Ostermiller
  • 99.4k
  • 18
  • 141
  • 364

Google has never limited itself to crawling and indexing just URLs that are in the sitemap. Such functionality does not exist, and I doubt that it ever will.

Sitemaps are fairly useless. They don't help with rankings. They rarely get Google to index pages it wouldn't otherwise index. Google really only uses them to choose preferred URLs, to specify alternate language URLs, and to give you extra data in search console. See The Sitemap Paradox.

You probably don't want to use robots.txt to disallow the URLs either. robots.txt blocks crawling but not indexing. You need to have Google re-crawl the URLs and see that they are gone. Googlebot needs to be able to access the URLs for that.

To clean up your hacked URLs, make sure they now return 404 status. Google will remove each of them within 24 hours of next crawling them. It could take Google a few months to remove all the URLs because it may not re-crawl some of them again soon. See Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

If there are not too many of the URLs, you can submit them individually through the Google Search Console Remove URLs Tool. That will get Google to remove them much faster than waiting around for the re-crawl, but there is no bulk remove feature.

Google has never limited itself to crawling and indexing just URLs that are in the sitemap. Such functionality does not exist, and I doubt that it ever will.

Sitemaps are fairly useless. They don't help with rankings. They rarely get Google to index pages it wouldn't otherwise index. Google really only uses them to choose preferred URLs, to specify alternate language URLs, and to give you extra data in search console. See The Sitemap Paradox.

Google has never limited itself to crawling and indexing just URLs that are in the sitemap. Such functionality does not exist, and I doubt that it ever will.

Sitemaps are fairly useless. They don't help with rankings. They rarely get Google to index pages it wouldn't otherwise index. Google really only uses them to choose preferred URLs, to specify alternate language URLs, and to give you extra data in search console. See The Sitemap Paradox.

You probably don't want to use robots.txt to disallow the URLs either. robots.txt blocks crawling but not indexing. You need to have Google re-crawl the URLs and see that they are gone. Googlebot needs to be able to access the URLs for that.

To clean up your hacked URLs, make sure they now return 404 status. Google will remove each of them within 24 hours of next crawling them. It could take Google a few months to remove all the URLs because it may not re-crawl some of them again soon. See Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

If there are not too many of the URLs, you can submit them individually through the Google Search Console Remove URLs Tool. That will get Google to remove them much faster than waiting around for the re-crawl, but there is no bulk remove feature.

Source Link
Stephen Ostermiller
  • 99.4k
  • 18
  • 141
  • 364

Google has never limited itself to crawling and indexing just URLs that are in the sitemap. Such functionality does not exist, and I doubt that it ever will.

Sitemaps are fairly useless. They don't help with rankings. They rarely get Google to index pages it wouldn't otherwise index. Google really only uses them to choose preferred URLs, to specify alternate language URLs, and to give you extra data in search console. See The Sitemap Paradox.