Skip to main content
31 votes

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404 A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped ...
MrWhite's user avatar
  • 42.9k
15 votes
Accepted

How can I use robots.txt to disallow subdomain only?

You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in ....
MrWhite's user avatar
  • 42.9k
14 votes

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

I'm going to answer the 2nd question. I was wondering if there is a quick way other than to manually remove these URLs from the google webmaster tools? https://developers.google.com/webmasters/...
pastepotpete's user avatar
13 votes

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

In short: you can't. Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html ...
cusco's user avatar
  • 131
12 votes
Accepted

Allow a folder and disallow all sub folders in robots.txt

User-agent: * Allow: /news/$ Disallow: /news/ Explanation: Google's robots.txt spec (https://developers.google.com/search/reference/robots_txt), which is more up to date than the "official" spec, ...
Henry Visotski's user avatar
11 votes

When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?

Your suspicions are right, with the exact reasoning that you mentioned. If you disallow robots from accessing the old site, they won't be able to see the 301 redirect.
Theo van der Zee's user avatar
10 votes
Accepted

How to resolve Google "Indexed, though blocked by robots.txt"

Google isn't crawling your page, but it is indexing the URL. It isn't indexing the content of the page, just the URL itself, possibly along with anchor text of links that point to it. Google says: ...
Stephen Ostermiller's user avatar
10 votes
Accepted

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

Incorrectly used meta tags Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two. <meta name="...
Stephen Ostermiller's user avatar
9 votes
Accepted

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

Google includes uncrawlable pages in the index when they are linked from other sites. That means that a link to the website like<a href="domain.com/en">[CHINESE] - Hey nighthawk</a> can ...
Stephen Ostermiller's user avatar
9 votes
Accepted

What if robots.txt disallows itself?

Robots.txt directives don't apply to robots.txt itself. Crawlers may fetch robots.txt even if it disallows itself. It is actually very common for robots.txt to disallow itself. Many websites ...
Stephen Ostermiller's user avatar
8 votes
Accepted

How do I "noindex" a text (.txt) file?

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. ...
Stephen Ostermiller's user avatar
8 votes
Accepted

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek ...
Maximillian Laumeister's user avatar
7 votes

What is a minimum valid robots.txt file?

The best minimal robots.txt is a completely empty file. Any other "null" directives such as an empty Disallow or Allow: * are not only useless because they are no-ops, but add unneeded complexity. ...
Maximillian Laumeister's user avatar
7 votes

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt: User-agent: MJ12bot Disallow: /
Sjoerd Linders's user avatar
7 votes

How do I edit robots.txt on a site hosted with Google Sites?

The robots.txt file is located at https://sites.google.com/robots.txt. You don't have access to edit that file, it is automatically generated by Google. The article you are referring to is ...
norcal johnny's user avatar
7 votes

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

robots.txt prevents pages from being specifically unindexable You read that right. Make the page crawlable and unindexable To make sure a page does not appear in Google search results, make sure it is ...
Tim Grant's user avatar
  • 529
7 votes

URLs with 'NoIndex` in robots.txt are being indexed by Google

"noindex" directives should not be used in your robots.txt file, instead a noindex meta tag should be added to any pages that you don't want indexed in Google. A NOINDEX tag looks like the below and ...
Matthew Elshaw's user avatar
7 votes

Should the "admin-ajax.php " file in my WordPress site be allow or disallowed in robots.txt?

This file is often used front-end by themes and plugins. It was changed in WordPress robots.txt file a couple of years ago to allow crawling for this reason, after an issue with google being unable to ...
Bronwyn V's user avatar
  • 3,423
7 votes

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

Not only can you not ensure it, it's highly likely that the link will be retrieved by programs not under the user's control. Many spam and phishing filters will pre-fetch any web pages linked to in ...
Mark's user avatar
  • 236
7 votes
Accepted

Do SEO analysis bots have any advantage in terms of website traffic?

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic ...
Matthew Edgar's user avatar
7 votes
Accepted

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/ This does look like a mistake in the Google docs IMO (as you suggest) for several reasons: That example ...
MrWhite's user avatar
  • 42.9k
6 votes

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I tried the robots.txt method and it didn't work. So I contacted the website on their email [email protected]: Hello, Can you please remove my personal website dimitarnestorov.com from your ...
Dimitar Nestorov's user avatar
6 votes

How to properly (dis)allow the archive.org bot? Did things change, if so when?

There are really 2 issues here: Will the robots.txt on your site Disallow (block) Wayback from crawling your site. Will Wayback crawl your site. For point #1: As others have said, the correct entry ...
Kevin Fegan's user avatar
6 votes
Accepted

What's wrong with my robots.txt file to block Google?

If you don't want to allowed to crawl your website on Google search result, then use this robots.txt User-agent: * Disallow: / These will block all directories. But if many website links to your ...
Goyllo's user avatar
  • 5,507
6 votes
Accepted

How to block only Yandex bot using robots.txt

There are many "Yandex bots". If you want to block all the Yandex bots then: User-agent: Yandex Disallow: / However, if you only want to block the main indexing "YandexBot" then: User-agent: ...
MrWhite's user avatar
  • 42.9k
6 votes

When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?

I believe no because the crawlers won't be able to crawl the old site to see the 301 status codes. Yes, exactly - this is not the "right move". If you implement a 301 redirect, presumably to preserve ...
DocRoot's user avatar
  • 4,297
6 votes

URLs with 'NoIndex` in robots.txt are being indexed by Google

Note that Noindex is not part of the original robots.txt specification. Google supported it as experimental feature (see: How does “Noindex:” in robots.txt work?), but it’s not clear if that is still ...
unor's user avatar
  • 21.8k
6 votes
Accepted

Is it good practice to block crawling of a website's privacy policy with robots.txt?

Your arguments against not blocking the indexing of these legal pages are valid. Matt Cuts did a video about this very topic here in 2009: https://www.youtube.com/watch?v=unEML5n5vTo While it's not ...
termsfeed's user avatar
  • 720
6 votes
Accepted

How to tell bots to forget a website and reindex it from scratch

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
Stephen Ostermiller's user avatar
6 votes
Accepted

Why are bots repeatedly crawling pages of my site that I never created?

Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site. An old site These ...
Stephen Ostermiller's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible