Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

49 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
103 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
3
votes
0answers
37 views

Should I block the '_next' directory in a site using Next.js with robots.txt?

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx ...
3
votes
0answers
17 views

Prevent crawling of mobile site will impact the actual's site ranking or indexing?

We have a responsive mobile-first indexing site in place. Unfortunately, the m.example.com - mobile version is also there and it ranks for certain keywords. We don't want to have such a separate m....
3
votes
0answers
408 views

Robots.txt checker says the files are allowed, but Search Console says they're blocked

All wp-content and wp-include files are blocked according to Google Search Console, and Googlebot can't crawl any JavaScript contained in those folders. GSC insists that these folders are blocked by ...
2
votes
1answer
61 views

Avoid unnecessary crawling of ad landing URLs by disallowing query string with robots.txt

Will the following configuration allow me to run an Ads campaign for products with the ?string parameter in the URL? User-agent: * Disallow: /*?string User-agent: AdsBot-Google Allow: / User-agent: ...
2
votes
1answer
76 views

What is a robots.txt file and how can I use it

I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create ...
2
votes
0answers
39 views

Is “cbsearch.site” a web robot, and if yes, how do I block it?

I saw invalid traffic in my Google AdSense account. I read some articles from the web, which all made it clear that invalid traffic could be caused by web "robots". So I tried to read on how ...
2
votes
0answers
29 views

Could allowing our images to be indexed hurt our web search rankings?

We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
2
votes
0answers
112 views

Restrict Google from indexing subdomain from Azure CDN services

How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
2
votes
0answers
123 views

In robots.txt - what are the directives from WBCPBlocker

On this site's robots.txt there is a comment preceding many rows of directives blocking SEO tool bots and was wondering if someone could decipher it for me. There's the standard User-agent: * Allow:...
2
votes
0answers
272 views

Google Blocked Resources showing incorrect data

In Google Webmaster Blocked Resources tab it is showing 500+ blocked resource (like .js and .css), but when I tested those URL in robots.txt Tester it is showing Allowed, then why not Google bot is ...
2
votes
0answers
277 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
2
votes
0answers
171 views

Google search console reports URLs blocked by robots.txt even though they shouldn't be

Recently I sent a sitemap to Google Search Console and I got some warnings saying that some urls are blocked by the robots.txt. I checked it and there is no single line that blocks the corresponding ...
2
votes
0answers
360 views

How to make nginx override web app route for sitemap and robots?

I would like to make nginx serve sitemap and robots. My problem is that route http://example.com/ is handled by flask, configured to handle request with certain patterns - http://example.com/sitemap....
2
votes
0answers
103 views

Google Webmaster Tools reports URLs blocked by robots.txt on my AVS tube website

I added a tube (with AVS script) in Google Webmaster Tools, but the issue is with the robots.txt file which is blocking everything. Here is my AVS tube robots.txt: User-agent: * Allow: / Disallow: /...
2
votes
0answers
289 views

SEO: Deindex member profile pages?

I have a forum where users can register and automatically a profile page is created, like here in Stackexchange. The problem with these pages is that sometimes users creates accounts at different ...
1
vote
0answers
40 views

Disallow dynamic URL paths ending in a common suffix using robots.txt

I want to block Google from crawling the following URLs: https://www.example.com/tonga/posts https://www.example.com/india/posts https://www.example.com/usa/posts https://www.example.com/uae/posts ...
1
vote
0answers
24 views

Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap

I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
1
vote
1answer
118 views

Google is indexing URLs with parameters that are disallowed in robots.txt despite canonical URLs without parameters listed in the sitemap

All of my webpages are showing ?mode=grid & ?mode=list in the google coverage. But submitted sitemap shows normal URLs. For example: example.com/page/?mode=grid example.com/page/?mode=list ...
1
vote
0answers
28 views

Google say URL is blocked by robots.txt but it isn't

My Google search console says that my submitted URL is blocked by robots.txt, but I checked header, tags and robots.txt it isn't. My URL was on #1 for my target keyword and daily get 1000 clicks ...
1
vote
0answers
47 views

Permanently redirecting (`301`) robots.txt from root (`/`) to static files directory (`/static/file/robots.txt`)

After reading up on this Google Webmasters article on creating a robots.txt file, I noticed that they mentioned that the robots.txt file should always be at the root directory (/). However, I've been ...
1
vote
0answers
26 views

Can I use a wildcard rule to prevent Yandex from indexing all tag and date URLs created by Joomla K2?

I have a site based on joomla and K2 component. I use K2 Tags and Calendar, but i have problem with pages indexing When yandex.ru begin to index it i find out that it index my tags and SEO test sites ...
1
vote
0answers
58 views

Google Search Console reports Soft 404 on an Excel file

since mid of this month, an Excel file that used to be indexed in Google Search is reported as a Soft 404 on Search Console File: https://neoschronos.com/assets/lean-canvas.xls We have multiple ...
1
vote
0answers
26 views

Why is Google indexing our robots.txt file and showing it in search results?

For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below. Our robots.txt file is not linked from anywhere on the site ...
1
vote
0answers
49 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
1
vote
0answers
108 views

Google is trying to index the mail.mydomain.example domain and says this is a coverage issue

I have sub domains for my domains. mail.mydomain.example, ftp.mydomain.example, etc. These sub domains are configured with the DNS server for the domains. Without doing anything on my part Google ...
1
vote
0answers
232 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
147 views

Prevent Indexing of Images from my website but hosted on S3 / Cloudinary

I understand that as a website/domain owner (eg: example.com), we can prevent images from our website from being indexed by Google (GoogleBot / GoogleBot-Image). We could add a meta tag to the whole ...
1
vote
0answers
183 views

How to resolve my webmaster crawl issue? URLs unreachable

Issue: When we tested a sample of the URLs from your Sitemap, we found that some of the URLs were unreachable. Please check your web server for possible misconfiguration, as these errors may be caused ...
1
vote
0answers
527 views

Use robots.txt to allow Google to crawl images but not index them

I have problem with on page optimization. I checked my site to see if it is compatibility for mobile devices with Google's tool. It showed me that page speed was low. Google also says it cannot ...
1
vote
0answers
96 views

Search Console - lock in robots.txt

In search console when I use "see like Google" I got this: resources not on my site are blocked by robots. How to fix this?
1
vote
0answers
711 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
1
vote
0answers
591 views

Magento Robots.txt and blocking cached images

We have a Magento based store. Images are resized and served cached dynamically. When the image cache in Magento is cleared, there are generated new paths and files. And I get 404 errors for them ...
1
vote
0answers
46 views

Canonical link or block in robots.txt for duplicate content across domains?

I've built a WordPress multisite for a dealership which has different franchises, ie Honda and BMW. Each franchise has its own website which have a used section, with identical content on them all for ...
1
vote
0answers
111 views

Bing is not crawling and indexing my mobile subdomain

BACKGROUND I have a mobile site located at a sub-domain of m.hydro-cleansing.com. The site is working fine and can be accessed. I am using a search API provided by Bing that should perform a site ...
1
vote
1answer
32 views

Block short URL in robots.txt

We have example.com/article-page-one/ This page has a short URL in Wordpress like this : example.com/?p=1 In this page we put short URL for users. What will happen if we block short URL? The robots....
1
vote
2answers
70 views

Will it effect SEO if there are 30 links on a page where the destination is blocked by robots.txt

I have a webpage on which there are around 30 data points which link to 30 different pages on my own site which are non-crawlable and have been blocked via robots.txt My question is if we use href ...
0
votes
0answers
25 views

Google Sitemap XML isn't showing About Page, Contact Page, Etc, only

My google sitemap has been uploaded to Google Search Console using example.com/sitemap_index.xml When I click to the page in google search console 'sitemaps read:' I can only see the following: ...
0
votes
0answers
17 views

Should API urls be disallowed in robots.text on SPAs

I have a single page app that I want google to index. However, I can see from the search console that Google also crawls /api/* urls. Search console does not tell wether google bot calls them directly ...
0
votes
0answers
18 views

how to specify all values for a parameter in robots.txt and sitemap.xml?

I have a php file that extracts a post id from a url's parameter to show a specific post content. for example it uses the postid parameter from the following url to show the post content with id of 21:...
0
votes
0answers
55 views

Google search says No information is available for my GitHub page

Today I search my GitHub page in Google Chrome and Firefox (Through Google search) and for the first time, it says no information is available (it says 2019- GitHub under that it says no information ...
0
votes
0answers
22 views

Interesting, but weird Google behavior on redirected URLs, if redirect targets are disallowed for crawling

There are two sites: a.example and b.example. URLs a.example/cat were allowed for crawling at a.example/robots.txt. URLs a.example/cat/page were disallowed for crawling at a.example/robots.txt. Then ...
0
votes
0answers
42 views

robots.txt query string disallow except for one case

I can't find solution for my task. For example, I have 2 pages: / and /some-category/. Both of them have GET parameters: page and several filters. I need the result: /, /?page=x, /some-category/ to be ...
0
votes
0answers
84 views

Google Search console reports home page blocked by robots.txt and search results only show the domain name even though it isn't blocked

For the last few days I haven't seen the title and meta description in Google search results. When I inspected URL in search console, I got this error message: URL is on Google, but has issues. ...
0
votes
0answers
162 views

Soft error 404 and submitted URL is blocked by Robots.txt

Soft error 404 and robots txt error:- we have recently submitted sitemap xml to Search console, but we got a warning stating 131 of the page links are softerror 404 and for the home site it says ...
0
votes
0answers
299 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
0
votes
0answers
86 views

Google reports that robots.txt is blocking site - but I don't think it is

I have a site map at http://www.example.com/sitemap_index.xml that has been sumbitted to Google Webmaster Tools, however... Google is reporting that my sitemap is being blocked by robots.txt. My ...
-1
votes
1answer
19 views

Why is Azure pinging my site?

I have noticed more and more, traffic from IPs assigned to Microsoft Azure/Microsoft Corporation hitting my WordPress site many times a day. It's direct traffic to just the home and privacy policy ...
-1
votes
1answer
680 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...