Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

0
votes
1answer
14 views

Should I block an .htm resource in robots.txt

We use scripts to create static .htm files to show content on our site. Mainly in an effort to reduce database calls. For example, a script that builds a file every 5 minutes that contains a list of ...
0
votes
1answer
49 views

Problem with Google Mobile Test - Blocking images

I have a situation when I check my Drupal 7 website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 56 resources ...
0
votes
1answer
97 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
-2
votes
1answer
271 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...
1
vote
0answers
29 views

Magento Robots.txt and blocking cached images

We have a Magento based store. Images are resized and served cached dynamically. When the image cache in Magento is cleared, there are generated new paths and files. And I get 404 errors for them ...
1
vote
0answers
47 views

Google mobile friendly resources which are blocked by robots.txt

I have a situation when I check my website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 5 resources which are ...
1
vote
0answers
44 views

Configuring the document root on a cookieless domain

I am configuring a cookieless domain for all my static content, I see a lot of articles and how-to's use the same document root for as their main domain. Why? This seems like a disadvantage to me, ...
1
vote
0answers
31 views

Canonical link or block in robots.txt for duplicate content across domains?

I've built a WordPress multisite for a dealership which has different franchises, ie Honda and BMW. Each franchise has its own website which have a used section, with identical content on them all for ...
1
vote
0answers
53 views

Bing is not crawling and indexing my mobile subdomain

BACKGROUND I have a mobile site located at a sub-domain of m.hydro-cleansing.com. The site is working fine and can be accessed. I am using a search API provided by Bing that should perform a site ...
0
votes
0answers
28 views

can't remove blocks on my resources in google webmaster tools

I received an alert from Google Webmaster Tools a couple weeks ago that stated: "Google systems have recently detected an issue with your homepage that affects how well our algorithms render and ...
0
votes
0answers
52 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
0
votes
0answers
84 views

Google webmaster tool Severe health issues are found in your property

I'm getting this message in GWT : Severe health issues are found in your property , Some important page is blocked by robots.txt. My robots.txt User-agent: * Disallow: /wp-admin/ Sitemap: ...
0
votes
0answers
28 views

SEO: Deindex member profile pages?

I have a forum where users can register and automatically a profile page is created, like here in Stackexchange. The problem with these pages is that sometimes users creates accounts at different ...
0
votes
0answers
33 views

How to design different web application environments with respect to SEO and server load?

Let's say I have a nice and valuable domain (somedomain.com) and I would like to develop a web application for this domain. I will host this web application on the cloud which I use currently for ...
0
votes
0answers
466 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...