Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
5
votes
1answer
103 views

Google CSE has indexed robots.txt itself

Google CSE has indexed robots.txt and now if someone searches for 'txt' it returns the robots.txt file which is really not ideal (as this is a bog-standard Drupal site, the string robots.txt actually ...
4
votes
4answers
729 views

No index for an entire site

I am finishing a new project. I have been working local for the last weeks/months. Now I want to begin the first tests on the real server and sql database. I do not want the site to be public yet and ...
1
vote
0answers
142 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
2
votes
1answer
603 views

How do I remove a large number of “Excluded” links in Google Search Console?

As you can see in the screenshot below, my site has a lot of "excluded" links, more than 34K, which is unusual: However I don't have that many links in my website: I'm worried about my SEO and ...
6
votes
1answer
238 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
2
votes
3answers
99 views

Can I block crawling specified images

The home page of my website has some images of users, I want to know if I can stop the crawling of these images with robots.txt Disallow: /img/users My question is I want to know if that can make a ...
1
vote
2answers
54 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
1
vote
0answers
69 views

Prevent Indexing of Images from my website but hosted on S3 / Cloudinary

I understand that as a website/domain owner (eg: example.com), we can prevent images from our website from being indexed by Google (GoogleBot / GoogleBot-Image). We could add a meta tag to the whole ...
0
votes
1answer
39 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
1answer
289 views

Google Search Console Crawl 404 linked Error from strange Website [duplicate]

Image: The error is displayed because the URL has added the symbol ; at the end. /wp-content/uploads/2018/06/iuiuiuiuiuiui.pdf; (incorrect) If we remove the symbol ; the URL it works normally. /...
1
vote
1answer
620 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
3
votes
0answers
68 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
0
votes
1answer
197 views

My robot.txt file looks fine but doesn't work

For my website I tried to make a robot.txt file so my SEO would be better. After adding the robot.txt to my website I expected it to work like it should, but when I checked my SEO with an online site, ...
2
votes
1answer
26 views

For documents that get generated with a new version id, how should I handle the old URLs? Robots.txt, 301 redirect, or create permanent URLs?

The site I am working on has around 800 products, some of these can have up to 2000 PDF and CAD drawings each. The links to these PDF and CAD files are combined into a single object and then linked ...
1
vote
2answers
41 views

Old dummy pages that should not be indexed are blocked by robots.txt, but Google is still indexing them. What now?

We have a rebuild new website and we have removed all the old pages (dummy pages comes with template) and blocked them via robots txt. now we keep receiving Index coverage issues warnings. Is it safe ...
1
vote
1answer
684 views

What is the ideal WordPress robots.txt for the best results? [closed]

Until now I've had my robots.txt file configured as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php As this has been some time now since I last touched it, I've been ...
2
votes
1answer
37 views

Not found page blocked via robots.txt still gets click/impression

Some dummy pages comes with template had been indexed early. Those are completely removed and blocked via robots.txt two month ago but we can still see those pages showing under Search Analytics in ...
1
vote
0answers
140 views

How to resolve my webmaster crawl issue? URLs unreachable

Issue: When we tested a sample of the URLs from your Sitemap, we found that some of the URLs were unreachable. Please check your web server for possible misconfiguration, as these errors may be caused ...
1
vote
0answers
40 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
1
vote
3answers
792 views

URL blocked by robots.txt warning while submitting sitemap in Google Search Console

I was submitting my sitemap in Google Webmasters and got this message when I clicked on the "test sitemap" button: -Warning - URL blocked by robots.txt. When I located my robots.txt file, this is ...
2
votes
2answers
407 views

Robots.txt to disallow on multi level subdomains

I've got a server, where I have multiple development projects for clients. While developing, each instance lives on client.dev.example.com. I need to stop crawlers from crawling the development ...
1
vote
1answer
412 views

Site appears blocked in robots, but Google Search Console still says any tested URL is allowed?

Working on a client's site, I noticed an error when trying to use a duplicate content checker tool. The tool stated that the home page was blocked by robots.txt. In search console, I can see the ...
2
votes
1answer
79 views

Can I have both “disallow nothing” and “disallow something” statements in my robots.txt?

I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ? ...
0
votes
1answer
48 views

Google doesn't send me to correct robots.txt directory

So I want to set a robots.txt in my site. Pretty straight forward place in root directory and good. But Google spits out a 404 error telling me the robots.txt is not in the place it expects. Not ...
3
votes
1answer
432 views

Bing stopped indexing the site after disallowing all crawlers except Googlebot, Yahoo-slurp, and Msnbot

Bing removed my top pages from results, 3 days ago they were showing at first page. Maybe i updated robots.txt file few weeks ago. Can someone check below template if there is something wrong that is ...
0
votes
1answer
193 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
21 views

How do instruct Google to index one of two similar domains with slightly different content?

If a website is available on two different domains, ex: domain1.com and domain2.com and the content of domain2.com differs slightly from that of domain1.com, how can I instruct Google to complete ...
3
votes
0answers
374 views

Robots.txt checker says the files are allowed, but Search Console says they're blocked

All wp-content and wp-include files are blocked according to Google Search Console, and Googlebot can't crawl any JavaScript contained in those folders. GSC insists that these folders are blocked by ...
1
vote
1answer
211 views

Do robots.txt and sitemap.xml need to be physical files? [closed]

I have both setup in my routes: Route::get('/robots.txt', function() { // robots.txt contents here }); Route::get('/sitemap.xml', function() { // sitemap.xml contents here }); I can access ...
2
votes
1answer
364 views

Can I disallow JS scripts in robots.txt for Googlebot if the website degrades to offer equivelant non-JS functionality?

To my knowledge, Googlebot is currently fully capable of rendering complex SPA applications, and this is recommended as a rule of thumb. The website is currently routed from server side and served as ...
1
vote
2answers
62 views

Will Google penalize me for loading images that it can't see? (blocked by robots.txt)

I have a script setup to load images for the visitor on another server. The robots.txt file on that server has the image path directory blocked by bots. Which means that Google can't see the images ...
1
vote
1answer
2k views

Do you need to add 301 redirect rules into robots.txt for search engines when you specify them in .htaccess?

Do I need to add 301 redirects to the robots.txt file? I will be adding the redirects to the .htaccess file, but not sure if the search engines need them in the robots.txt file as well to prevent any ...
2
votes
1answer
27 views

Does a “Disallow:” rule with nothing following it in robots.txt block my entire domain?

If my website's robots.txt has the following code, does that mean my entire site is being blocked from indexing, or do I need to append the code with a /? User-Agent: * Disallow:
2
votes
1answer
856 views

Clean up hacked site by getting Google to crawl and index only the URLs in the sitemap

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites. So we tried playing with ...
4
votes
1answer
2k views

Allow a folder and disallow all sub folders in robots.txt

I would like to allow folder /news/ and disallow all the sub folders under /news/ e.g. /news/abc/, /news/123/. How can I do that please? I think Disallow: /news/ will block everything in it, ...
3
votes
1answer
82 views

Will a dynamic robots.txt file that disallows crawling based on the time of day hurt SEO?

We have a serious traffic issue on our site and we want to eliminate crawlers as part of the problem. If we disallow crawling using robots.txt only during some hours, will it hurt our SEO?
1
vote
0answers
417 views

Use robots.txt to allow Google to crawl images but not index them

I have problem with on page optimization. I checked my site to see if it is compatibility for mobile devices with Google's tool. It showed me that page speed was low. Google also says it cannot ...
3
votes
2answers
1k views

Is it good practice to block crawling of a website's privacy policy with robots.txt?

I'm not sure whether I should add my website's privacy policy to my robots.txt file or not. I want to follow good SEO practice. I have no reason to hide my privacy policy. Arguments against: I want ...
1
vote
2answers
845 views

Robots.txt isn't preventing my site from being crawled

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites. ...
1
vote
0answers
94 views

Search Console - lock in robots.txt

In search console when I use "see like Google" I got this: resources not on my site are blocked by robots. How to fix this?
3
votes
1answer
641 views

How to disallow hash fragments in robots.txt file

I have a similar URL in my website that I want to stop all robots from crawling /review/dir/dir/dir/dir/#review-form The rules I've tried are: Disallow:/review/*review-form$ Disallow:/*\#review-form$...
4
votes
1answer
9k views

Should the “admin-ajax.php ” file in my WordPress site be allow or disallowed in robots.txt?

My WordPress site in cPanel is serving the admin-ajax.php file which is wasting CPU resources. How can I solve this problem? Should I use this in my robots.txt file or not? User-agent: * Disallow: /...
1
vote
1answer
161 views

URL is indexed but description not available due to “this site's robots.txt”

I have a similar URL indexed by Google https://example.com/companies/company?utm_source=xxx And this output: A description for this result is not available because of this site's robots.txt. Here ...
1
vote
1answer
102 views

How to stop indexing/crawling for Shop Checkout Summaries?

I have a small shop checkout that uses cookies for my cart and after the payment is done it generates a unique order-id creates an "overview" page with an url like this: http://localhost:8180/app_dev....
1
vote
0answers
103 views

How to prevent CDN content URLs being indexed by Google [duplicate]

Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too). So, even when a URL is disallowed in robots....
1
vote
1answer
60 views

Does Alexa's crawler need a robots.txt on each sub domain?

I have a robots.txt on my www site. Do I also need one on other sites like my app and forums sites? Alexa audit keeps finding bad links, missing tags, and other "errors" on my vanilla forums site ...
1
vote
2answers
588 views

How do you add a rule just for a specific bot to robots.txt?

I have a small website, for which the current robots.txt looks like so: User-agent: * Disallow: Sitemap: https://www.myawesomesite.com/sitemap.xml Now, I want to add another rule like so (with the ...
3
votes
2answers
2k views

URLs with 'NoIndex` in robots.txt are being indexed by Google

In my robots.txt file (http://www.tutorvista.com/robots.txt), I'm using Noindex: /content/... to disallow indexing: This should mean that http://www.tutorvista.com/content/ and anything below this ...
0
votes
1answer
69 views

I'm receiving an error about my robots.txt in Google Search Console [closed]

My website in Google Search Console is reporting an error: You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a ...