Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

1
vote
0answers
17 views

How to allow only admin-ajax.php to be crawled? [on hold]

My robots.txt looks as follows: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Google Webmaster Tool recently raised a warning saying: Indexed, though blocked by robots.txt I ...
0
votes
0answers
5 views

Sitemap on Google Search Console not helping top 6 links to change [duplicate]

I am currently attempting to alter the way Google will display the top 6 links when you search for our URL on Google's search engine. I have read about creating a sitemap.xml file and created the file ...
2
votes
2answers
37 views

Disallow URLs with an ID from being crawled

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example....
1
vote
1answer
21 views

Main Website with Sub Directory = 2 Robots & Sitemaps respectively?

I have a website example.com and a subdirectory with example.com/blog/. (I can consider it as two websites) The main website example.com is the service based and it is not a WordPress site. So, I ...
0
votes
0answers
28 views

Soft error 404 and submitted URL is blocked by Robots.txt

Soft error 404 and robots txt error:- we have recently submitted sitemap xml to Search console, but we got a warning stating 131 of the page links are softerror 404 and for the home site it says ...
1
vote
1answer
23 views

Magento robots.txt rule also blocking JavaScript from WordPress

I have a Magento site and a WordPress site is used in a directory for the blog. My blog keeps getting hit as not mobile friendly and the reason I believe is that there are certain resources blocked in ...
1
vote
1answer
35 views

Yandex ignores Crawl-delay?

I try to get the Yandex Bot under control and I'm close to block it. After 2 months of robots.txt adjustments, it still keeps hammering my server from multiple IPs. User-Agent: * Crawl-delay: 4 ...
1
vote
1answer
41 views

Robot.Txt File interpretation problem?

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages: {block name="frontend_robots_txt_user_agent"} User-agent: * {/block} {block name="...
1
vote
1answer
42 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
0
votes
1answer
26 views

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory, Is this possible or I must change robots.txt for drafts starting as draft:myDraft?
1
vote
0answers
37 views

Google is trying to index the mail.mydomain.example domain and says this is a coverage issue

I have sub domains for my domains. mail.mydomain.example, ftp.mydomain.example, etc. These sub domains are configured with the DNS server for the domains. Without doing anything on my part Google ...
4
votes
1answer
52 views

Robots.txt allow directory root while disallowing all deeper URLs

I would like to allow a path and disallow everything else in the directory. User-Agent: * Allow: /example/ Disallow: /example/* Will this work? If not what would be a better way of doing this?
0
votes
2answers
63 views

Disallowing URL parameter in robots.txt

I have important URLs on my site to which I’d like to append temporary parameters. For example: example.com/tool/input will have the parameter example.com/tool/input?foo input in the URL is ...
1
vote
1answer
28 views

Google desktop crawler showing unreachable - mobile crawler is working

I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this ...
1
vote
0answers
23 views

Canoical vs robots disallow for sorted content [duplicate]

I have a web page that displays 5 items (each with a headline, image and short content like in a microblog). Item 1 Item 2 ... Item 5 Based on the query parameter in the URL, I will be sorting the ...
0
votes
1answer
23 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
0
votes
2answers
125 views

How to configure the robots.txt file with a multilingual site?

I have a site with Drupal 8 and I activate English and French (Language from the URL path prefix). Here is what my url resembles : https://www.example.com/en/node/add https://www.example.com/fr/node/...
1
vote
1answer
61 views

Should tag and author pages be disallowed in robots.txt?

According to some articles that talk about crawl budget, we should decrease the number pages crawled and provide the only important and valuable pages to search engine crawlers. We have 3,000 ...
1
vote
1answer
54 views

Hacked site created foreign language tags pages, can we block them in robots.txt to remove them from Google?

Our WordPress website was hacked which is now revamped and updated to the WP v5.0.3 but still Tags and 35fc4a-50164960-421910 or 6d285b-31601402-365308 type URLs are indexed in Google. Can we block ...
1
vote
1answer
36 views

Prevent dynamically generated thumbnails from being crawled

I have been using Raven Tools, and I want these dynamically generated thumbnails to be not crawled or not be flagged as No TITLE images. I tried adding X-Robots-Tag "noindex, nofollow" on each image ...
6
votes
1answer
778 views

How to tell bots to forget a website and reindex it from scratch

I was wondering if there was anything I could add to robots.txt or sitemap to tell bots to completely forget everything they know about a website and index it from scratch? Context: After replacing a ...
2
votes
1answer
31 views

Can I use robots.txt to allow Google to access some folders, but disallow everything else?

I want to allow Google bots to crawl only my index.php and the content of some folders. Would that work? User-agent: * Allow: /index*.php Allow: /folder1/ Allow: /folder2/ Allow: /folder3/ Allow: /...
1
vote
1answer
40 views

How to prevent pages from being indexed in Google Search by using robots.txt? [duplicate]

In my robots.txt I disallowed some of the "thank you" pages on my site that I don't want to indexed by Google or be in Google search results. But those pages have recently become searchable on Google. ...
1
vote
1answer
100 views

How to add robots.txt to my WordPress site?

How to add robots.txt to my wordpress.com site?
1
vote
1answer
83 views

Google can't access my page due to robots.txt but the testing tool says it's OK

My robots.txt is the typical that allows all bots everywhere User-agent: * Disallow: / When I test it with Google tool everything is OK. But in Google Search Console I get an error that my site can'...
4
votes
1answer
685 views

Google Search Console says “submitted URL blocked by robots.txt” even though there is no robots.txt file

Today I got a email notice from Google that there were search problems with my site. I logged into Google Search Console. It shows that 1 file was being blocked by robots.txt, and has been for ...
0
votes
1answer
89 views

Pages uploaded to WordPress as media are causing mobile usability errors in Google Search Console [duplicate]

the new search console is pointing out usability errors on mobile devices on WordPress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
2
votes
1answer
215 views

How do I disallow some but not all subdirectories of a folder using robots.txt?

I have one folder named Frontend and under this folder I have subfolder names. Like: Images CSS JS Font I want to disallow below folder: /forntend/css/ /forntend/js/ /forntend/font/ but I don't ...
4
votes
1answer
136 views

Will forbidding “bad bot” user agents via .htaccess add a layer of protection against email address scraping beyond the existing captcha?

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot ...
0
votes
2answers
69 views

Major Crawlers not honoring robots.txt?

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site. I cleaned the websites in question and updated the robots.txt. the ...
2
votes
1answer
49 views

What kind of pages are suggested to be disallowed in robots.txt? [closed]

I tried googling, but did not find a good resource. What are some guidelines about deciding what to disallow in robots.txt?
3
votes
3answers
832 views

We have been adding 404 Not Found pages to robots.txt, but now Google is indexing them

Index coverage says "Indexed, though blocked by robots.txt". Those pages are no longer active and return 404 status. So that they don't get crawled and indexed, our SEO guys have been adding them to ...
0
votes
1answer
30 views

what is url robot search engine when enter to my website [closed]

what is url search engine robots when enter to my website for example i add this code in .htaccess this code just allow google robots read the robots.txt i need to add url ( msn , yahoo , bing ) ...
3
votes
1answer
10k views

How to resolve Google “Indexed, though blocked by robots.txt”

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
5
votes
1answer
86 views

Google CSE has indexed robots.txt itself

Google CSE has indexed robots.txt and now if someone searches for 'txt' it returns the robots.txt file which is really not ideal (as this is a bog-standard Drupal site, the string robots.txt actually ...
4
votes
4answers
514 views

No index for an entire site

I am finishing a new project. I have been working local for the last weeks/months. Now I want to begin the first tests on the real server and sql database. I do not want the site to be public yet and ...
1
vote
0answers
121 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
2
votes
1answer
499 views

How do I remove a large number of “Excluded” links in Google Search Console?

As you can see in the screenshot below, my site has a lot of "excluded" links, more than 34K, which is unusual: However I don't have that many links in my website: I'm worried about my SEO and ...
6
votes
1answer
220 views

What if robots.txt disallows itself?

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
2
votes
3answers
91 views

Can I block crawling specified images

The home page of my website has some images of users, I want to know if I can stop the crawling of these images with robots.txt Disallow: /img/users My question is I want to know if that can make a ...
1
vote
2answers
51 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
1
vote
0answers
65 views

Prevent Indexing of Images from my website but hosted on S3 / Cloudinary

I understand that as a website/domain owner (eg: example.com), we can prevent images from our website from being indexed by Google (GoogleBot / GoogleBot-Image). We could add a meta tag to the whole ...
0
votes
1answer
38 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
1answer
279 views

Google Search Console Crawl 404 linked Error from strange Website [duplicate]

Image: The error is displayed because the URL has added the symbol ; at the end. /wp-content/uploads/2018/06/iuiuiuiuiuiui.pdf; (incorrect) If we remove the symbol ; the URL it works normally. /...
1
vote
1answer
582 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
3
votes
0answers
63 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
0
votes
1answer
174 views

My robot.txt file looks fine but doesn't work

For my website I tried to make a robot.txt file so my SEO would be better. After adding the robot.txt to my website I expected it to work like it should, but when I checked my SEO with an online site, ...
2
votes
1answer
26 views

For documents that get generated with a new version id, how should I handle the old URLs? Robots.txt, 301 redirect, or create permanent URLs?

The site I am working on has around 800 products, some of these can have up to 2000 PDF and CAD drawings each. The links to these PDF and CAD files are combined into a single object and then linked ...
1
vote
2answers
38 views

Old dummy pages that should not be indexed are blocked by robots.txt, but Google is still indexing them. What now?

We have a rebuild new website and we have removed all the old pages (dummy pages comes with template) and blocked them via robots txt. now we keep receiving Index coverage issues warnings. Is it safe ...