31 votes

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404 A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped ...
MrWhite's user avatar
  • 42.7k
20 votes
Accepted

Are web crawlers able to find a secondary robots.txt in a sub-directory?

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of ...
Ilmari Karonen's user avatar
15 votes
Accepted

How can I use robots.txt to disallow subdomain only?

You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in ....
MrWhite's user avatar
  • 42.7k
14 votes

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

I'm going to answer the 2nd question. I was wondering if there is a quick way other than to manually remove these URLs from the google webmaster tools? https://developers.google.com/webmasters/...
pastepotpete's user avatar
13 votes

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

In short: you can't. Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html ...
cusco's user avatar
  • 131
11 votes

When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?

Your suspicions are right, with the exact reasoning that you mentioned. If you disallow robots from accessing the old site, they won't be able to see the 301 redirect.
Theo van der Zee's user avatar
10 votes
Accepted

Allow a folder and disallow all sub folders in robots.txt

User-agent: * Allow: /news/$ Disallow: /news/ Explanation: Google's robots.txt spec (https://developers.google.com/search/reference/robots_txt), which is more up to date than the "official" spec, ...
Henry Visotski's user avatar
10 votes
Accepted

How to resolve Google "Indexed, though blocked by robots.txt"

Google isn't crawling your page, but it is indexing the URL. It isn't indexing the content of the page, just the URL itself, possibly along with anchor text of links that point to it. Google says: ...
Stephen Ostermiller's user avatar
10 votes
Accepted

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

Incorrectly used meta tags Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two. <meta name="...
Stephen Ostermiller's user avatar
9 votes

Are indexed Wordpress admin pages a security threat?

When a new bug is found, in this case in wordpress, the first thing hackers will do is try to find vulnerable sites that use wordpress. A good way of doing it is trying to find wp-admin pages on ...
CristianTM's user avatar
9 votes
Accepted

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

Google includes uncrawlable pages in the index when they are linked from other sites. That means that a link to the website like<a href="domain.com/en">[CHINESE] - Hey nighthawk</a> can ...
Stephen Ostermiller's user avatar
9 votes
Accepted

What if robots.txt disallows itself?

Robots.txt directives don't apply to robots.txt itself. Crawlers may fetch robots.txt even if it disallows itself. It is actually very common for robots.txt to disallow itself. Many websites ...
Stephen Ostermiller's user avatar
8 votes

Are indexed Wordpress admin pages a security threat?

Security issue? Doubt it. It just makes your site more visible to people who likes to attack wordpress sites. Other than the visibility everything is the same, security through obscurity is not ...
cdvv7788's user avatar
  • 189
8 votes
Accepted

How do I "noindex" a text (.txt) file?

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. ...
Stephen Ostermiller's user avatar
8 votes
Accepted

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek ...
Maximillian Laumeister's user avatar
7 votes

What is a minimum valid robots.txt file?

The best minimal robots.txt is a completely empty file. Any other "null" directives such as an empty Disallow or Allow: * are not only useless because they are no-ops, but add unneeded complexity. ...
Maximillian Laumeister's user avatar
7 votes

Are indexed Wordpress admin pages a security threat?

Google doesn't want to index non-content pages including admin pages and pages that ask you to login. Putting wp-admin in robots.txt is better for SEO because it will usually prevent the page from ...
Stephen Ostermiller's user avatar
7 votes

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt: User-agent: MJ12bot Disallow: /
Sjoerd Linders's user avatar
7 votes

How do I edit robots.txt on a site hosted with Google Sites?

The robots.txt file is located at https://sites.google.com/robots.txt. You don't have access to edit that file, it is automatically generated by Google. The article you are referring to is ...
norcal johnny's user avatar
7 votes

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

robots.txt prevents pages from being specifically unindexable You read that right. Make the page crawlable and unindexable To make sure a page does not appear in Google search results, make sure it is ...
Tim Grant's user avatar
  • 529
7 votes

URLs with 'NoIndex` in robots.txt are being indexed by Google

"noindex" directives should not be used in your robots.txt file, instead a noindex meta tag should be added to any pages that you don't want indexed in Google. A NOINDEX tag looks like the below and ...
Matthew Elshaw's user avatar
7 votes

Should the "admin-ajax.php " file in my WordPress site be allow or disallowed in robots.txt?

This file is often used front-end by themes and plugins. It was changed in WordPress robots.txt file a couple of years ago to allow crawling for this reason, after an issue with google being unable to ...
Bronwyn V's user avatar
  • 3,423
7 votes
Accepted

How to tell bots to forget a website and reindex it from scratch

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
Stephen Ostermiller's user avatar
7 votes

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

Not only can you not ensure it, it's highly likely that the link will be retrieved by programs not under the user's control. Many spam and phishing filters will pre-fetch any web pages linked to in ...
Mark's user avatar
  • 236
7 votes
Accepted

Do SEO analysis bots have any advantage in terms of website traffic?

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic ...
Matthew Edgar's user avatar
7 votes
Accepted

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/ This does look like a mistake in the Google docs IMO (as you suggest) for several reasons: That example ...
MrWhite's user avatar
  • 42.7k
6 votes

How to properly (dis)allow the archive.org bot? Did things change, if so when?

There are really 2 issues here: Will the robots.txt on your site Disallow (block) Wayback from crawling your site. Will Wayback crawl your site. For point #1: As others have said, the correct entry ...
Kevin Fegan's user avatar
6 votes

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

This will not affect your SEO. Any web crawler will be able to find a website's HTML sitemap as long as there is a link pointing to it. Remember that you can submit your sitemap to most search engines ...
Raul Reyes's user avatar
  • 2,486
6 votes
Accepted

What's wrong with my robots.txt file to block Google?

If you don't want to allowed to crawl your website on Google search result, then use this robots.txt User-agent: * Disallow: / These will block all directories. But if many website links to your ...
Goyllo's user avatar
  • 5,507
6 votes
Accepted

How to block only Yandex bot using robots.txt

There are many "Yandex bots". If you want to block all the Yandex bots then: User-agent: Yandex Disallow: / However, if you only want to block the main indexing "YandexBot" then: User-agent: ...
MrWhite's user avatar
  • 42.7k

Only top scored, non community-wiki answers of a minimum length are eligible