31
votes
Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?
My website was hacked which is now recovered but the hacker indexed 5000 URLs in Google and now I get error 404
A 404 is probably preferable to blocking with robots.txt if you want these URLs dropped ...
- 43k
15
votes
Accepted
How can I use robots.txt to disallow subdomain only?
You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in ....
- 43k
14
votes
Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?
I'm going to answer the 2nd question.
I was wondering if there is a quick way other than to manually remove
these URLs from the google webmaster tools?
https://developers.google.com/webmasters/...
- 156
13
votes
How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?
In short: you can't.
Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html ...
- 131
12
votes
Accepted
Allow a folder and disallow all sub folders in robots.txt
User-agent: *
Allow: /news/$
Disallow: /news/
Explanation:
Google's robots.txt spec (https://developers.google.com/search/reference/robots_txt), which is more up to date than the "official" spec, ...
- 5,246
11
votes
When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?
Your suspicions are right, with the exact reasoning that you mentioned. If you disallow robots from accessing the old site, they won't be able to see the 301 redirect.
- 1,319
10
votes
Accepted
How to resolve Google "Indexed, though blocked by robots.txt"
Google isn't crawling your page, but it is indexing the URL. It isn't indexing the content of the page, just the URL itself, possibly along with anchor text of links that point to it. Google says:
...
- 99.2k
10
votes
Accepted
How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?
Incorrectly used meta tags
Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two.
<meta name="...
- 99.2k
9
votes
Accepted
Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German
Google includes uncrawlable pages in the index when they are linked from other sites.
That means that a link to the website like<a href="domain.com/en">[CHINESE] - Hey nighthawk</a> can ...
- 99.2k
9
votes
Accepted
What if robots.txt disallows itself?
Robots.txt directives don't apply to robots.txt itself. Crawlers may fetch robots.txt even if it disallows itself.
It is actually very common for robots.txt to disallow itself. Many websites ...
- 99.2k
8
votes
Accepted
How do I "noindex" a text (.txt) file?
The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. ...
- 99.2k
8
votes
Accepted
Why does it cause issues to use robots.txt and a meta robots tag at the same time?
The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek ...
- 16.2k
7
votes
Do I really have to block MJ12Bot (as the prevailing visitor on my site)?
MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt:
User-agent: MJ12bot
Disallow: /
- 171
7
votes
What is a minimum valid robots.txt file?
The best minimal robots.txt is a completely empty file.
Any other "null" directives such as an empty Disallow or Allow: * are not only useless because they are no-ops, but add unneeded complexity.
...
- 16.2k
7
votes
How do I edit robots.txt on a site hosted with Google Sites?
The robots.txt file is located at https://sites.google.com/robots.txt.
You don't have access to edit that file, it is automatically generated by Google.
The article you are referring to is ...
- 2,400
7
votes
Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German
robots.txt prevents pages from being specifically unindexable
You read that right.
Make the page crawlable and unindexable
To make sure a page does not appear in Google search results, make sure it is ...
- 529
7
votes
URLs with 'NoIndex` in robots.txt are being indexed by Google
"noindex" directives should not be used in your robots.txt file, instead a noindex meta tag should be added to any pages that you don't want indexed in Google.
A NOINDEX tag looks like the below and ...
7
votes
Should the "admin-ajax.php " file in my WordPress site be allow or disallowed in robots.txt?
This file is often used front-end by themes and plugins.
It was changed in WordPress robots.txt file a couple of years ago to allow crawling for this reason, after an issue with google being unable to ...
- 3,433
7
votes
How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?
Not only can you not ensure it, it's highly likely that the link will be retrieved by programs not under the user's control. Many spam and phishing filters will pre-fetch any web pages linked to in ...
- 236
7
votes
Accepted
Do SEO analysis bots have any advantage in terms of website traffic?
If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic ...
- 1,471
7
votes
Should I block the '_next' directory in a site using Next.js with robots.txt?
I wrote an entire blog post on the topic:
Robots.txt - Block _Next Folder in Next.js Application
I tried to block the entire /_next/ directory in my robots.txt file to test out the idea.
I then ran ...
- 171
7
votes
Accepted
Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)
the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/
This does look like a mistake in the Google docs IMO (as you suggest) for several reasons:
That example ...
- 43k
6
votes
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I tried the robots.txt method and it didn't work. So I contacted the website on their email [email protected]:
Hello,
Can you please remove my personal website dimitarnestorov.com from
your ...
- 161
6
votes
Accepted
What's wrong with my robots.txt file to block Google?
If you don't want to allowed to crawl your website on Google search result, then use this robots.txt
User-agent: *
Disallow: /
These will block all directories.
But if many website links to your ...
- 5,507
6
votes
Accepted
How to block only Yandex bot using robots.txt
There are many "Yandex bots". If you want to block all the Yandex bots then:
User-agent: Yandex
Disallow: /
However, if you only want to block the main indexing "YandexBot" then:
User-agent: ...
- 43k
6
votes
When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?
I believe no because the crawlers won't be able to crawl the old site to see the 301 status codes.
Yes, exactly - this is not the "right move". If you implement a 301 redirect, presumably to preserve ...
- 4,297
6
votes
URLs with 'NoIndex` in robots.txt are being indexed by Google
Note that Noindex is not part of the original robots.txt specification. Google supported it as experimental feature (see: How does “Noindex:” in robots.txt work?), but it’s not clear if that is still ...
- 21.8k
6
votes
Accepted
Is it good practice to block crawling of a website's privacy policy with robots.txt?
Your arguments against not blocking the indexing of these legal pages are valid. Matt Cuts did a video about this very topic here in 2009:
https://www.youtube.com/watch?v=unEML5n5vTo
While it's not ...
- 720
6
votes
Accepted
How to tell bots to forget a website and reindex it from scratch
That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience.
Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
- 99.2k
6
votes
Accepted
Why are bots repeatedly crawling pages of my site that I never created?
Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site.
An old site
These ...
- 99.2k
Only top scored, non community-wiki answers of a minimum length are eligible