Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

0
votes
0answers
28 views

Robots are still Crawling Website after Disallowing Using robot.txt and Redirecting it Https from Http using .htaccess [closed]

I have tried to disallow robots to crawling my website by adding robot.txt and redirecting website to https as Explained below .htacces Contains the following code:- RewriteEngine on RewriteCond %{...
0
votes
2answers
37 views

Major Crawlers not honoring robots.txt?

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site. I cleaned the websites in question and updated the robots.txt. the ...
2
votes
1answer
39 views

What kind of pages are suggested to be disallowed in robots.txt? [closed]

I tried googling, but did not find a good resource. What are some guidelines about deciding what to disallow in robots.txt?
3
votes
3answers
92 views

We have been adding 404 Not Found pages to robots.txt, but now Google is indexing them

Index coverage says "Indexed, though blocked by robots.txt". Those pages are no longer active and return 404 status. So that they don't get crawled and indexed, our SEO guys have been adding them to ...
0
votes
1answer
24 views

what is url robot search engine when enter to my website [closed]

what is url search engine robots when enter to my website for example i add this code in .htaccess this code just allow google robots read the robots.txt i need to add url ( msn , yahoo , bing ) ...
1
vote
1answer
30 views

How to resolve Google “Indexed, though blocked by robots.txt”

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
0
votes
0answers
23 views

Do i need to add robot.txt file in sitemap?

In google new search console, we are fixing (Indexed, not submitted in sitemap) issues. In that category robots.txt file taken place. Do i really need to add robot.txt file in sitemap?
4
votes
1answer
50 views

Google CSE has indexed robots.txt itself

Google CSE has indexed robots.txt and now if someone searches for 'txt' it returns the robots.txt file which is really not ideal (as this is a bog-standard Drupal site, the string robots.txt actually ...
3
votes
2answers
44 views

No index for an entire site

I am finishing a new project. I have been working local for the last weeks/months. Now I want to begin the first tests on the real server and sql database. I do not want the site to be public yet and ...
1
vote
0answers
22 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
2
votes
1answer
57 views

How do I remove a large number of “Excluded” links in Google Search Console?

As you can see in the screenshot below, my site has a lot of "excluded" links, more than 34K, which is unusual: However I don't have that many links in my website: I'm worried about my SEO and ...
5
votes
1answer
187 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
0
votes
0answers
40 views

Embed code content: Indexed, though blocked by robots.txt

I maintain a site that presents some pages to users, like mysite.com/articles/abc123. I also provide embed codes so that a thumbnail/link to each article can be embedded in other sites (similar to ...
2
votes
3answers
51 views

Can I block crawling specified images

The home page of my website has some images of users, I want to know if I can stop the crawling of these images with robots.txt Disallow: /img/users My question is I want to know if that can make a ...
1
vote
2answers
38 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
1
vote
0answers
24 views

Prevent Indexing of Images from my website but hosted on S3 / Cloudinary

I understand that as a website/domain owner (eg: example.com), we can prevent images from our website from being indexed by Google (GoogleBot / GoogleBot-Image). We could add a meta tag to the whole ...
0
votes
1answer
29 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
1answer
171 views

Google Search Console Crawl 404 linked Error from strange Website [duplicate]

Image: The error is displayed because the URL has added the symbol ; at the end. /wp-content/uploads/2018/06/iuiuiuiuiuiui.pdf; (incorrect) If we remove the symbol ; the URL it works normally. /...
0
votes
0answers
118 views

Google Search Console reports widgets.pinterest.com is a blocked resource, even after I removed that code from my pages

I added Pinterest widgets to my website (on WordPress). I included a meta tag for confirmation. But now google webmaster displays that 2512 pages of the site have blocked resources from Pinterest. I ...
1
vote
1answer
173 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
0
votes
0answers
16 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
3
votes
0answers
41 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
0
votes
1answer
86 views

My robot.txt file looks fine but doesn't work

For my website I tried to make a robot.txt file so my SEO would be better. After adding the robot.txt to my website I expected it to work like it should, but when I checked my SEO with an online site, ...
2
votes
1answer
23 views

For documents that get generated with a new version id, how should I handle the old URLs? Robots.txt, 301 redirect, or create permanent URLs?

The site I am working on has around 800 products, some of these can have up to 2000 PDF and CAD drawings each. The links to these PDF and CAD files are combined into a single object and then linked ...
1
vote
2answers
32 views

Old dummy pages that should not be indexed are blocked by robots.txt, but Google is still indexing them. What now?

We have a rebuild new website and we have removed all the old pages (dummy pages comes with template) and blocked them via robots txt. now we keep receiving Index coverage issues warnings. Is it safe ...
1
vote
1answer
130 views

What is the ideal WordPress robots.txt for the best results? [closed]

Until now I've had my robots.txt file configured as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php As this has been some time now since I last touched it, I've been ...
2
votes
1answer
33 views

Not found page blocked via robots.txt still gets click/impression

Some dummy pages comes with template had been indexed early. Those are completely removed and blocked via robots.txt two month ago but we can still see those pages showing under Search Analytics in ...
1
vote
0answers
76 views

How to resolve my webmaster crawl issue? URLs unreachable

Issue: When we tested a sample of the URLs from your Sitemap, we found that some of the URLs were unreachable. Please check your web server for possible misconfiguration, as these errors may be caused ...
1
vote
0answers
29 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
1
vote
3answers
352 views

URL blocked by robots.txt warning while submitting sitemap in Google Search Console

I was submitting my sitemap in Google Webmasters and got this message when I clicked on the "test sitemap" button: -Warning - URL blocked by robots.txt. When I located my robots.txt file, this is ...
2
votes
2answers
170 views

Robots.txt to disallow on multi level subdomains

I've got a server, where I have multiple development projects for clients. While developing, each instance lives on client.dev.example.com. I need to stop crawlers from crawling the development ...
1
vote
1answer
297 views

Site appears blocked in robots, but Google Search Console still says any tested URL is allowed?

Working on a client's site, I noticed an error when trying to use a duplicate content checker tool. The tool stated that the home page was blocked by robots.txt. In search console, I can see the ...
2
votes
1answer
66 views

Can I have both “disallow nothing” and “disallow something” statements in my robots.txt?

I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ? ...
0
votes
1answer
40 views

Google doesn't send me to correct robots.txt directory

So I want to set a robots.txt in my site. Pretty straight forward place in root directory and good. But Google spits out a 404 error telling me the robots.txt is not in the place it expects. Not ...
0
votes
0answers
12 views

Loosing 90% of my indexed pages and increasing every day! [duplicate]

I have well over 10000 pages indexed and slowly but surely it has dwindled to almost nothing. When it got to 5000 pages being indexed I wasn't too worried, however I am soon going to have none? ...
4
votes
1answer
184 views

Bing stopped indexing the site after disallowing all crawlers except Googlebot, Yahoo-slurp, and Msnbot

Bing removed my top pages from results, 3 days ago they were showing at first page. Maybe i updated robots.txt file few weeks ago. Can someone check below template if there is something wrong that is ...
0
votes
1answer
97 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
16 views

How do instruct Google to index one of two similar domains with slightly different content?

If a website is available on two different domains, ex: domain1.com and domain2.com and the content of domain2.com differs slightly from that of domain1.com, how can I instruct Google to complete ...
2
votes
0answers
128 views

Robots.txt checker says the files are allowed, but Search Console says they're blocked

All wp-content and wp-include files are blocked according to Google Search Console, and Googlebot can't crawl any JavaScript contained in those folders. GSC insists that these folders are blocked by ...
1
vote
1answer
125 views

Do robots.txt and sitemap.xml need to be physical files? [closed]

I have both setup in my routes: Route::get('/robots.txt', function() { // robots.txt contents here }); Route::get('/sitemap.xml', function() { // sitemap.xml contents here }); I can access ...
2
votes
1answer
126 views

Can I disallow JS scripts in robots.txt for Googlebot if the website degrades to offer equivelant non-JS functionality?

To my knowledge, Googlebot is currently fully capable of rendering complex SPA applications, and this is recommended as a rule of thumb. The website is currently routed from server side and served as ...
1
vote
2answers
40 views

Will Google penalize me for loading images that it can't see? (blocked by robots.txt)

I have a script setup to load images for the visitor on another server. The robots.txt file on that server has the image path directory blocked by bots. Which means that Google can't see the images ...
1
vote
1answer
467 views

Do you need to add 301 redirect rules into robots.txt for search engines when you specify them in .htaccess?

Do I need to add 301 redirects to the robots.txt file? I will be adding the redirects to the .htaccess file, but not sure if the search engines need them in the robots.txt file as well to prevent any ...
2
votes
1answer
27 views

Does a “Disallow:” rule with nothing following it in robots.txt block my entire domain?

If my website's robots.txt has the following code, does that mean my entire site is being blocked from indexing, or do I need to append the code with a /? User-Agent: * Disallow:
2
votes
1answer
267 views

Clean up hacked site by getting Google to crawl and index only the URLs in the sitemap

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites. So we tried playing with ...
3
votes
1answer
728 views

Allow a folder and disallow all sub folders in robots.txt

I would like to allow folder /news/ and disallow all the sub folders under /news/ e.g. /news/abc/, /news/123/. How can I do that please? I think Disallow: /news/ will block everything in it, ...
3
votes
1answer
62 views

Will a dynamic robots.txt file that disallows crawling based on the time of day hurt SEO?

We have a serious traffic issue on our site and we want to eliminate crawlers as part of the problem. If we disallow crawling using robots.txt only during some hours, will it hurt our SEO?
1
vote
0answers
272 views

Use robots.txt to allow Google to crawl images but not index them

I have problem with on page optimization. I checked my site to see if it is compatibility for mobile devices with Google's tool. It showed me that page speed was low. Google also says it cannot ...
2
votes
2answers
272 views

Is it good practice to block crawling of a website's privacy policy with robots.txt?

I'm not sure whether I should add my website's privacy policy to my robots.txt file or not. I want to follow good SEO practice. I have no reason to hide my privacy policy. Arguments against: I want ...
1
vote
2answers
389 views

Robots.txt isn't preventing my site from being crawled

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites. ...