So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites.

So we tried playing with robots.txt i.e.:

User-agent: *


Sitemap: http://www.example.com/sitemap.xml

But when I enter the bad URL in robots.txt tester, it still allow the URL that we don't want.

Is there any way that Google only crawls the sitemap on robots.txt without manually entering all the bad links on the Disallow?

1 Answer 1


Google has never limited itself to crawling and indexing just URLs that are in the sitemap. Such functionality does not exist, and I doubt that it ever will.

Sitemaps are fairly useless. They don't help with rankings. They rarely get Google to index pages it wouldn't otherwise index. Google really only uses them to choose preferred URLs, to specify alternate language URLs, and to give you extra data in search console. See The Sitemap Paradox.

You probably don't want to use robots.txt to disallow the URLs either. robots.txt blocks crawling but not indexing. You need to have Google re-crawl the URLs and see that they are gone. Googlebot needs to be able to access the URLs for that.

To clean up your hacked URLs, make sure they now return 404 status. Google will remove each of them within 24 hours of next crawling them. It could take Google a few months to remove all the URLs because it may not re-crawl some of them again soon. See Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

If there are not too many of the URLs, you can submit them individually through the Google Search Console Remove URLs Tool. That will get Google to remove them much faster than waiting around for the re-crawl, but there is no bulk remove feature.

  • You helped me with my sitemap question @StephenOstermiller. Am I OK to ask more about your above answer in comments in order to expand the answer for other readers? I don't understand why having a sitemap.xml is so vital for search engine submission to the point where it's an instant fail if one is not present but, as you said, they are largely ignored and bots do whatever they want. Apr 4, 2022 at 8:51
  • Lack of an XML sitemap is far from an "instant fail." I only have a sitemap for my largest site with millions of URLs. My twenty smaller sites do great with SEO without sitemaps. Sitemaps are easy enough to implement that they usually get recommended. But their SEO benefits are very limited. They can get Googlebot to come crawl all your URLs, presence of URLs in the sitemap is a canonical signal to Google, and they can give you extra data in Google Search Console. They don't help with indexing or ranking, and they can't limit crawling. Apr 4, 2022 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.