All Questions
35 questions
11
votes
2
answers
1k
views
What bots are really worth letting onto a site?
Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site?
My first ...
- 2,080
10
votes
1
answer
2k
views
How does "Noindex:" in robots.txt work?
I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.
Disallow: /page-one.html
...
- 99.4k
9
votes
1
answer
7k
views
How can I stop Google from indexing embedded iframes?
I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
- 245
5
votes
4
answers
8k
views
Does Google ignore robots.txt [duplicate]
I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
- 153
5
votes
2
answers
2k
views
Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?
A lot of websites hide their sitemap.xml file.
What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file.
So my question is ...
- 689
4
votes
1
answer
1k
views
Link rel="nofollow" vs robots.txt disallow for "/search" URLs?
There's a /search path on my web app that I don't want robots to crawl or index.
What would be the best approach for this scenario?
1: Block them from accessing by adding a Disallow for /search on ...
- 988
4
votes
2
answers
721
views
Controlling robot crawling concurrency using robots.txt and the undocumented setting "host-load"
I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
- 141
4
votes
1
answer
420
views
Is there a way to allow ad bots to crawl a website that has "noindex,nofollow" on it?
Due to duplicated content, there is a "noindex,nofollow" on the website, but in order to run ppc ads, there needs to be a way for the ads bots to crawl the website
Can this be done without ...
- 41
4
votes
1
answer
1k
views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
- 145
3
votes
3
answers
3k
views
Can web crawlers that visit my site fake their user agent?
I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest.
I want to know if I add User-agent: Googlebot, will fake ...
- 131
3
votes
1
answer
15k
views
How to block only Yandex bot using robots.txt
Can you show me how robots.txt will look when I block only Yandex bot?
I want to allow Googlebot and block Yandex bot.
- 381
3
votes
2
answers
5k
views
How to allow Googlebot and Bingbot to crawl mysite?
Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
- 33
3
votes
2
answers
5k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
- 102
3
votes
1
answer
199
views
Googlebot-Image incorrectly crossing domain URLs
My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com).
We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
- 133
3
votes
1
answer
948
views
Google bots and iframe
Here is the problem:
Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains:
Disallow: /article/
...
- 53
2
votes
2
answers
146
views
Disallow Crawling of All Search Pages
Using robots.txt I am attempting to stop all crawling of search URLs
Disallow: /rest_of_url/search&tour*
Above is what I am using. Our URL looks like the following for all search results. ...
- 253
2
votes
1
answer
476
views
Stop bots from crawling old links with extensions
I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist.
So I'm trying to find out if there is a ...
- 133
2
votes
1
answer
453
views
Allow all bots except Googlebot which should only be able to crawl the home page
I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else.
I'd prefer to do this in robots.txt. How can I ...
2
votes
2
answers
6k
views
redirecting bots and crawlers to another but not human via htaccess
I would to apply this diagram via htaccess I tried a lots of codes but I failed every time
So I need to redirecting bots and crawlers especially from facebook via .htaccess
- 29
2
votes
1
answer
966
views
How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?
I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
- 21
2
votes
1
answer
28
views
How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?
I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt
Could someone elaborate on the difference between these two:
http://example.com/page.htm
allow: /page
...
- 41
2
votes
2
answers
83
views
Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?
I've written my own SiteSearch Script in PHP.
The SiteSearch parses two GET parameters from the queryString:
search // the search-phrase
filters (optional) // indicating which parts of the site to ...
- 2,365
2
votes
1
answer
712
views
"Is robots.txt blocking important pages?" in Search Console regarding static folder
This is the contents of my robots.txt file:
User-agent: *
Disallow: /static/*
Disallow: /templates/*
Disallow: /translations/*
Sitemap: https://xx.xxx.com/sitemap.xml
However I don't know why I am ...
- 222
2
votes
1
answer
3k
views
Do I need to disallow mediapartners-google in robots.txt for SEO?
I found that most of the websites disallow mediapartners-google in robots.txt.
Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
- 155
2
votes
1
answer
250
views
robots.txt for website hosted in a subdirectory
I have 2 website which is hosted in a shared hosting.
1st website
example.com hosted in a root directory as /public_html/
2nd website
example2.com hosted in a sub-directory directory as /public_html/...
- 177
2
votes
1
answer
104
views
Blocking vs noindex to reduce crawl requests
I observed that GoogleBot is making a lot of duplicate requests for the same URLs from my website within a week. Amongst these requests a majority were for low/thin value pages(no or very low SERP,not ...
- 446
2
votes
1
answer
3k
views
In robots.txt, why would there be "Allow: /*" below a Disallow rule?
I want to allow full access to all Google
I am specially confused about Allow: /* below Disallow:
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google AdSense
User-agent: ...
2
votes
0
answers
433
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
0
answers
61
views
Disallow crawling of all links to a domain, except only specific URLs with parameters
I have a domain like example.com under which everything is blocked and links to an SPA.
Like example.com, and we've disabled everything like so:
User-agent: *
Disallow: /
Now this SPA uses parameters ...
- 121
1
vote
1
answer
193
views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
- 251
1
vote
1
answer
282
views
Do or can robots cause considerable performance issues?
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Our setup:
Lets say I have site www.mysite....
- 113
1
vote
1
answer
94
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
1
vote
0
answers
371
views
Stop getting traffic from Googlebot on high server load
In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing.
So what I thought of; is depending on the ...
- 111
0
votes
2
answers
167
views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress (...
- 113
0
votes
0
answers
25
views
Issues with Googlebot crawling and negative number pages
I am dealing with very heavy load from Googlebot, I have hundreds of sites running on multiple IPs all on the same server and googlebot is constantly crawling these sites causing a lot of availability ...
- 101