Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

1
vote
0answers
32 views

How do I remove a large number of “Excluded” links in Google Search Console?

As you can see in the screenshot below, my site has a lot of "excluded" links, more than 34K, which is unusual: However I don't have that many links in my website: I'm worried about my SEO and ...
2
votes
1answer
64 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
0
votes
0answers
7 views

Robots.txt help [closed]

Will the following disallow the directories listed and it's subdirectories and allow everything else?? User-agent: * Disallow: /wp-admin/ Disallow: /hirensbootcd/* Disallow: /ultimatebootcd/* ...
0
votes
0answers
7 views

URLs not indexed, but not being blocked by my robot.txt [closed]

My site is www.yohobed.com, in search console there are some URLs being showed as no index as they are being tagged as "nofollow". but there is no such thing as that. Can someone help me?
0
votes
0answers
29 views

Embed code content: Indexed, though blocked by robots.txt

I maintain a site that presents some pages to users, like mysite.com/articles/abc123. I also provide embed codes so that a thumbnail/link to each article can be embedded in other sites (similar to ...
2
votes
3answers
45 views

Can I block crawling specified images

The home page of my website has some images of users, I want to know if I can stop the crawling of these images with robots.txt Disallow: /img/users My question is I want to know if that can make a ...
1
vote
1answer
22 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
1
vote
0answers
19 views

Prevent Indexing of Images from my website but hosted on S3 / Cloudinary

I understand that as a website/domain owner (eg: example.com), we can prevent images from our website from being indexed by Google (GoogleBot / GoogleBot-Image). We could add a meta tag to the whole ...
0
votes
1answer
27 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
1answer
138 views

Google Search Console Crawl 404 linked Error from strange Website

Image: The error is displayed because the URL has added the symbol ; at the end. /wp-content/uploads/2018/06/iuiuiuiuiuiui.pdf; (incorrect) If we remove the symbol ; the URL it works normally. /...
0
votes
0answers
72 views

Google Search Console reports widgets.pinterest.com is a blocked resource, even after I removed that code from my pages

I added Pinterest widgets to my website (on WordPress). I included a meta tag for confirmation. But now google webmaster displays that 2512 pages of the site have blocked resources from Pinterest. I ...
1
vote
1answer
70 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
0
votes
0answers
16 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
3
votes
0answers
33 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
0
votes
1answer
71 views

My robot.txt file looks fine but doesn't work

For my website I tried to make a robot.txt file so my SEO would be better. After adding the robot.txt to my website I expected it to work like it should, but when I checked my SEO with an online site, ...
2
votes
1answer
22 views

For documents that get generated with a new version id, how should I handle the old URLs? Robots.txt, 301 redirect, or create permanent URLs?

The site I am working on has around 800 products, some of these can have up to 2000 PDF and CAD drawings each. The links to these PDF and CAD files are combined into a single object and then linked ...
1
vote
2answers
30 views

Old dummy pages that should not be indexed are blocked by robots.txt, but Google is still indexing them. What now?

We have a rebuild new website and we have removed all the old pages (dummy pages comes with template) and blocked them via robots txt. now we keep receiving Index coverage issues warnings. Is it safe ...
1
vote
1answer
67 views

What is the ideal WordPress robots.txt for the best results? [closed]

Until now I've had my robots.txt file configured as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php As this has been some time now since I last touched it, I've been ...
2
votes
1answer
31 views

Not found page blocked via robots.txt still gets click/impression

Some dummy pages comes with template had been indexed early. Those are completely removed and blocked via robots.txt two month ago but we can still see those pages showing under Search Analytics in ...
1
vote
0answers
43 views

How to resolve my webmaster crawl issue? URLs unreachable

Issue: When we tested a sample of the URLs from your Sitemap, we found that some of the URLs were unreachable. Please check your web server for possible misconfiguration, as these errors may be caused ...
1
vote
0answers
28 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
1
vote
3answers
198 views

URL blocked by robots.txt warning while submitting sitemap in Google Search Console

I was submitting my sitemap in Google Webmasters and got this message when I clicked on the "test sitemap" button: -Warning - URL blocked by robots.txt. When I located my robots.txt file, this is ...
2
votes
2answers
75 views

Robots.txt to disallow on multi level subdomains

I've got a server, where I have multiple development projects for clients. While developing, each instance lives on client.dev.example.com. I need to stop crawlers from crawling the development ...
1
vote
0answers
258 views

Site appears blocked in robots, but Google Search Console still says any tested URL is allowed?

Working on a client's site, I noticed an error when trying to use a duplicate content checker tool. The tool stated that the home page was blocked by robots.txt. In search console, I can see the ...
2
votes
1answer
65 views

Can I have both “disallow nothing” and “disallow something” statements in my robots.txt?

I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ? ...
0
votes
1answer
39 views

Google doesn't send me to correct robots.txt directory

So I want to set a robots.txt in my site. Pretty straight forward place in root directory and good. But Google spits out a 404 error telling me the robots.txt is not in the place it expects. Not ...
0
votes
0answers
12 views

Loosing 90% of my indexed pages and increasing every day! [duplicate]

I have well over 10000 pages indexed and slowly but surely it has dwindled to almost nothing. When it got to 5000 pages being indexed I wasn't too worried, however I am soon going to have none? ...
4
votes
1answer
113 views

Bing stopped indexing the site after disallowing all crawlers except Googlebot, Yahoo-slurp, and Msnbot

Bing removed my top pages from results, 3 days ago they were showing at first page. Maybe i updated robots.txt file few weeks ago. Can someone check below template if there is something wrong that is ...
0
votes
1answer
68 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
14 views

How do instruct Google to index one of two similar domains with slightly different content?

If a website is available on two different domains, ex: domain1.com and domain2.com and the content of domain2.com differs slightly from that of domain1.com, how can I instruct Google to complete ...
2
votes
0answers
77 views

Robots.txt checker says the files are allowed, but Search Console says they're blocked

All wp-content and wp-include files are blocked according to Google Search Console, and Googlebot can't crawl any JavaScript contained in those folders. GSC insists that these folders are blocked by ...
1
vote
1answer
103 views

Do robots.txt and sitemap.xml need to be physical files? [closed]

I have both setup in my routes: Route::get('/robots.txt', function() { // robots.txt contents here }); Route::get('/sitemap.xml', function() { // sitemap.xml contents here }); I can access ...
2
votes
1answer
84 views

Can I disallow JS scripts in robots.txt for Googlebot if the website degrades to offer equivelant non-JS functionality?

To my knowledge, Googlebot is currently fully capable of rendering complex SPA applications, and this is recommended as a rule of thumb. The website is currently routed from server side and served as ...
1
vote
2answers
40 views

Will Google penalize me for loading images that it can't see? (blocked by robots.txt)

I have a script setup to load images for the visitor on another server. The robots.txt file on that server has the image path directory blocked by bots. Which means that Google can't see the images ...
1
vote
1answer
247 views

Do you need to add 301 redirect rules into robots.txt for search engines when you specify them in .htaccess?

Do I need to add 301 redirects to the robots.txt file? I will be adding the redirects to the .htaccess file, but not sure if the search engines need them in the robots.txt file as well to prevent any ...
2
votes
1answer
26 views

Does a “Disallow:” rule with nothing following it in robots.txt block my entire domain?

If my website's robots.txt has the following code, does that mean my entire site is being blocked from indexing, or do I need to append the code with a /? User-Agent: * Disallow:
2
votes
1answer
183 views

Clean up hacked site by getting Google to crawl and index only the URLs in the sitemap

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites. So we tried playing with ...
3
votes
1answer
442 views

Allow a folder and disallow all sub folders in robots.txt

I would like to allow folder /news/ and disallow all the sub folders under /news/ e.g. /news/abc/, /news/123/. How can I do that please? I think Disallow: /news/ will block everything in it, ...
3
votes
1answer
60 views

Will a dynamic robots.txt file that disallows crawling based on the time of day hurt SEO?

We have a serious traffic issue on our site and we want to eliminate crawlers as part of the problem. If we disallow crawling using robots.txt only during some hours, will it hurt our SEO?
1
vote
0answers
186 views

Use robots.txt to allow Google to crawl images but not index them

I have problem with on page optimization. I checked my site to see if it is compatibility for mobile devices with Google's tool. It showed me that page speed was low. Google also says it cannot ...
2
votes
2answers
198 views

Is it good practice to block crawling of a website's privacy policy with robots.txt?

I'm not sure whether I should add my website's privacy policy to my robots.txt file or not. I want to follow good SEO practice. I have no reason to hide my privacy policy. Arguments against: I want ...
1
vote
2answers
264 views

Robots.txt isn't preventing my site from being crawled

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites. ...
1
vote
0answers
90 views

Search Console - lock in robots.txt

In search console when I use "see like Google" I got this: resources not on my site are blocked by robots. How to fix this?
2
votes
1answer
335 views

How to disallow hash fragments in robots.txt file

I have a similar URL in my website that I want to stop all robots from crawling /review/dir/dir/dir/dir/#review-form The rules I've tried are: Disallow:/review/*review-form$ Disallow:/*\#review-form$...
0
votes
0answers
39 views

Site's description not showing for robots.txt? [duplicate]

When I search on Google through WordPress site domain name without TLD, it shows the site's information but, in the Title section it shows only www.example.com, no title and description section like ...
3
votes
1answer
4k views

“admin-ajax.php ” file should allow or disallow in robots.txt file?

My Wordpress site c-panel showing admin-ajax.php file wasting more CPU resources. how to solve the problem. Should i use this command in robots.txt file or not - User-agent: * Disallow: /wp-admin/ ...
1
vote
1answer
160 views

URL is indexed but description not available due to “this site's robots.txt”

I have a similar URL indexed by Google https://example.com/companies/company?utm_source=xxx And this output: A description for this result is not available because of this site's robots.txt. Here ...
1
vote
1answer
63 views

How to stop indexing/crawling for Shop Checkout Summaries?

I have a small shop checkout that uses cookies for my cart and after the payment is done it generates a unique order-id creates an "overview" page with an url like this: http://localhost:8180/app_dev....
1
vote
0answers
28 views

How to prevent CDN content URLs being indexed by Google [duplicate]

Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too). So, even when a URL is disallowed in robots....
1
vote
1answer
40 views

Does Alexa's crawler need a robots.txt on each sub domain?

I have a robots.txt on my www site. Do I also need one on other sites like my app and forums sites? Alexa audit keeps finding bad links, missing tags, and other "errors" on my vanilla forums site ...