Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

1
vote
1answer
22 views

How to prevent pages from being indexed in Google Search by using robots.txt? [duplicate]

In my robots.txt I disallowed some of the "thank you" pages on my site that I don't want to indexed by Google or be in Google search results. But those pages have recently become searchable on Google. ...
1
vote
1answer
24 views

How to add robots.txt to my WordPress site?

How to add robots.txt to my wordpress.com site?
1
vote
1answer
26 views

Google can't access my page due to robots.txt but the testing tool says it's OK

My robots.txt is the typical that allows all bots everywhere User-agent: * Disallow: / When I test it with Google tool everything is OK. But in Google Search Console I get an error that my site can'...
4
votes
1answer
61 views

Google Search Console says “submitted URL blocked by robots.txt” even though there is no robots.txt file

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for ...
0
votes
1answer
29 views

Error in search console!

the new search console is pointing out usability errors on mobile devices on wordpress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
2
votes
1answer
42 views

How do I disallow some but not all subdirectories of a folder using robots.txt?

I have one folder named Frontend and under this folder I have subfolder names. Like: Images CSS JS Font I want to disallow below folder: /forntend/css/ /forntend/js/ /forntend/font/ but I don't ...
4
votes
0answers
58 views

Will forbidding “bad bot” user agents via .htaccess add a layer of protection against email address scraping beyond the existing captcha?

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot ...
0
votes
2answers
44 views

Major Crawlers not honoring robots.txt?

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site. I cleaned the websites in question and updated the robots.txt. the ...
2
votes
1answer
44 views

What kind of pages are suggested to be disallowed in robots.txt? [closed]

I tried googling, but did not find a good resource. What are some guidelines about deciding what to disallow in robots.txt?
3
votes
3answers
133 views

We have been adding 404 Not Found pages to robots.txt, but now Google is indexing them

Index coverage says "Indexed, though blocked by robots.txt". Those pages are no longer active and return 404 status. So that they don't get crawled and indexed, our SEO guys have been adding them to ...
0
votes
1answer
25 views

what is url robot search engine when enter to my website [closed]

what is url search engine robots when enter to my website for example i add this code in .htaccess this code just allow google robots read the robots.txt i need to add url ( msn , yahoo , bing ) ...
1
vote
1answer
381 views

How to resolve Google “Indexed, though blocked by robots.txt”

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
4
votes
1answer
58 views

Google CSE has indexed robots.txt itself

Google CSE has indexed robots.txt and now if someone searches for 'txt' it returns the robots.txt file which is really not ideal (as this is a bog-standard Drupal site, the string robots.txt actually ...
3
votes
2answers
81 views

No index for an entire site

I am finishing a new project. I have been working local for the last weeks/months. Now I want to begin the first tests on the real server and sql database. I do not want the site to be public yet and ...
1
vote
0answers
41 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
2
votes
1answer
111 views

How do I remove a large number of “Excluded” links in Google Search Console?

As you can see in the screenshot below, my site has a lot of "excluded" links, more than 34K, which is unusual: However I don't have that many links in my website: I'm worried about my SEO and ...
5
votes
1answer
189 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
0
votes
0answers
50 views

Embed code content: Indexed, though blocked by robots.txt

I maintain a site that presents some pages to users, like mysite.com/articles/abc123. I also provide embed codes so that a thumbnail/link to each article can be embedded in other sites (similar to ...
2
votes
3answers
57 views

Can I block crawling specified images

The home page of my website has some images of users, I want to know if I can stop the crawling of these images with robots.txt Disallow: /img/users My question is I want to know if that can make a ...
1
vote
2answers
44 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
1
vote
0answers
32 views

Prevent Indexing of Images from my website but hosted on S3 / Cloudinary

I understand that as a website/domain owner (eg: example.com), we can prevent images from our website from being indexed by Google (GoogleBot / GoogleBot-Image). We could add a meta tag to the whole ...
0
votes
1answer
29 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
1answer
208 views

Google Search Console Crawl 404 linked Error from strange Website [duplicate]

Image: The error is displayed because the URL has added the symbol ; at the end. /wp-content/uploads/2018/06/iuiuiuiuiuiui.pdf; (incorrect) If we remove the symbol ; the URL it works normally. /...
0
votes
0answers
163 views

Google Search Console reports widgets.pinterest.com is a blocked resource, even after I removed that code from my pages

I added Pinterest widgets to my website (on WordPress). I included a meta tag for confirmation. But now google webmaster displays that 2512 pages of the site have blocked resources from Pinterest. I ...
1
vote
1answer
308 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
0
votes
0answers
16 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
3
votes
0answers
48 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
0
votes
1answer
108 views

My robot.txt file looks fine but doesn't work

For my website I tried to make a robot.txt file so my SEO would be better. After adding the robot.txt to my website I expected it to work like it should, but when I checked my SEO with an online site, ...
2
votes
1answer
24 views

For documents that get generated with a new version id, how should I handle the old URLs? Robots.txt, 301 redirect, or create permanent URLs?

The site I am working on has around 800 products, some of these can have up to 2000 PDF and CAD drawings each. The links to these PDF and CAD files are combined into a single object and then linked ...
1
vote
2answers
32 views

Old dummy pages that should not be indexed are blocked by robots.txt, but Google is still indexing them. What now?

We have a rebuild new website and we have removed all the old pages (dummy pages comes with template) and blocked them via robots txt. now we keep receiving Index coverage issues warnings. Is it safe ...
1
vote
1answer
286 views

What is the ideal WordPress robots.txt for the best results? [closed]

Until now I've had my robots.txt file configured as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php As this has been some time now since I last touched it, I've been ...
2
votes
1answer
34 views

Not found page blocked via robots.txt still gets click/impression

Some dummy pages comes with template had been indexed early. Those are completely removed and blocked via robots.txt two month ago but we can still see those pages showing under Search Analytics in ...
1
vote
0answers
101 views

How to resolve my webmaster crawl issue? URLs unreachable

Issue: When we tested a sample of the URLs from your Sitemap, we found that some of the URLs were unreachable. Please check your web server for possible misconfiguration, as these errors may be caused ...
1
vote
0answers
31 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
1
vote
3answers
448 views

URL blocked by robots.txt warning while submitting sitemap in Google Search Console

I was submitting my sitemap in Google Webmasters and got this message when I clicked on the "test sitemap" button: -Warning - URL blocked by robots.txt. When I located my robots.txt file, this is ...
2
votes
2answers
244 views

Robots.txt to disallow on multi level subdomains

I've got a server, where I have multiple development projects for clients. While developing, each instance lives on client.dev.example.com. I need to stop crawlers from crawling the development ...
1
vote
1answer
308 views

Site appears blocked in robots, but Google Search Console still says any tested URL is allowed?

Working on a client's site, I noticed an error when trying to use a duplicate content checker tool. The tool stated that the home page was blocked by robots.txt. In search console, I can see the ...
2
votes
1answer
70 views

Can I have both “disallow nothing” and “disallow something” statements in my robots.txt?

I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ? ...
0
votes
1answer
40 views

Google doesn't send me to correct robots.txt directory

So I want to set a robots.txt in my site. Pretty straight forward place in root directory and good. But Google spits out a 404 error telling me the robots.txt is not in the place it expects. Not ...
0
votes
0answers
13 views

Loosing 90% of my indexed pages and increasing every day! [duplicate]

I have well over 10000 pages indexed and slowly but surely it has dwindled to almost nothing. When it got to 5000 pages being indexed I wasn't too worried, however I am soon going to have none? ...
4
votes
1answer
248 views

Bing stopped indexing the site after disallowing all crawlers except Googlebot, Yahoo-slurp, and Msnbot

Bing removed my top pages from results, 3 days ago they were showing at first page. Maybe i updated robots.txt file few weeks ago. Can someone check below template if there is something wrong that is ...
0
votes
1answer
119 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
17 views

How do instruct Google to index one of two similar domains with slightly different content?

If a website is available on two different domains, ex: domain1.com and domain2.com and the content of domain2.com differs slightly from that of domain1.com, how can I instruct Google to complete ...
2
votes
0answers
184 views

Robots.txt checker says the files are allowed, but Search Console says they're blocked

All wp-content and wp-include files are blocked according to Google Search Console, and Googlebot can't crawl any JavaScript contained in those folders. GSC insists that these folders are blocked by ...
1
vote
1answer
141 views

Do robots.txt and sitemap.xml need to be physical files? [closed]

I have both setup in my routes: Route::get('/robots.txt', function() { // robots.txt contents here }); Route::get('/sitemap.xml', function() { // sitemap.xml contents here }); I can access ...
2
votes
1answer
145 views

Can I disallow JS scripts in robots.txt for Googlebot if the website degrades to offer equivelant non-JS functionality?

To my knowledge, Googlebot is currently fully capable of rendering complex SPA applications, and this is recommended as a rule of thumb. The website is currently routed from server side and served as ...
1
vote
2answers
41 views

Will Google penalize me for loading images that it can't see? (blocked by robots.txt)

I have a script setup to load images for the visitor on another server. The robots.txt file on that server has the image path directory blocked by bots. Which means that Google can't see the images ...
1
vote
1answer
694 views

Do you need to add 301 redirect rules into robots.txt for search engines when you specify them in .htaccess?

Do I need to add 301 redirects to the robots.txt file? I will be adding the redirects to the .htaccess file, but not sure if the search engines need them in the robots.txt file as well to prevent any ...
2
votes
1answer
27 views

Does a “Disallow:” rule with nothing following it in robots.txt block my entire domain?

If my website's robots.txt has the following code, does that mean my entire site is being blocked from indexing, or do I need to append the code with a /? User-Agent: * Disallow:
2
votes
1answer
392 views

Clean up hacked site by getting Google to crawl and index only the URLs in the sitemap

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites. So we tried playing with ...