Skip to main content
use example.com, formatting, title clarity
Source Link
Stephen Ostermiller
  • 99.4k
  • 18
  • 141
  • 364

Crawling Clean up hacked site by getting Google to crawl and index only the URLs in the sitemap on google webmaster tools

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites.

So we tried playing with robots.txtrobots.txt i.e.:

User-agent: *

Disallow:

Sitemap: http://www.website.com/sitemap.xml

User-agent: *

Disallow: 

Sitemap: http://www.example.com/sitemap.xml

But when I enter the bad URL in robots.txtrobots.txt tester, it still allow the URL that we don't want.

Is there any way that googleGoogle only crawls the sitemap on robots.txt without manually entering all the bad links on the "Disallow"Disallow?

Crawling only the sitemap on google webmaster tools

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites.

So we tried playing with robots.txt i.e.:

User-agent: *

Disallow:

Sitemap: http://www.website.com/sitemap.xml

But when I enter the bad URL in robots.txt tester, it still allow the URL that we don't want.

Is there any way that google only crawls the sitemap on robots.txt without manually entering all the bad links on the "Disallow"?

Clean up hacked site by getting Google to crawl and index only the URLs in the sitemap

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites.

So we tried playing with robots.txt i.e.:

User-agent: *

Disallow: 

Sitemap: http://www.example.com/sitemap.xml

But when I enter the bad URL in robots.txt tester, it still allow the URL that we don't want.

Is there any way that Google only crawls the sitemap on robots.txt without manually entering all the bad links on the Disallow?

Source Link

Crawling only the sitemap on google webmaster tools

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites.

So we tried playing with robots.txt i.e.:

User-agent: *

Disallow:

Sitemap: http://www.website.com/sitemap.xml

But when I enter the bad URL in robots.txt tester, it still allow the URL that we don't want.

Is there any way that google only crawls the sitemap on robots.txt without manually entering all the bad links on the "Disallow"?