We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
4
votes
2answers
3k views

Prevent indexing of image or file

I understand that you can do the following to generally prevent images or pages from getting indexed. Add to the page's meta section: <meta name="robots" content="noindex" /> Edit the robots....
2
votes
1answer
168 views

robots.txt proper way to set it up

If you want to allow all search engines to crawl a site and only block one specific folder, is this correct? User-agent: * Disallow: /folder_name/ If you don't want to block any folder then is this ...
2
votes
1answer
633 views

“Is robots.txt blocking important pages?” in Search Console regarding static folder

This is the contents of my robots.txt file: User-agent: * Disallow: /static/* Disallow: /templates/* Disallow: /translations/* Sitemap: https://xx.xxx.com/sitemap.xml However I don't know why I am ...
2
votes
1answer
99 views

robots.txt for website hosted in a subdirectory

I have 2 website which is hosted in a shared hosting. 1st website example.com hosted in a root directory as /public_html/ 2nd website example2.com hosted in a sub-directory directory as /public_html/...
0
votes
2answers
183 views

Spider-Trap on a GitHub Site

I have a GitHub site and I hate web-crawlers that disobey or ignore robots.txt. How would I set up a Spider-Trap on a GitHub site that the robots.txt disallows the trap, so the good bots aren't ...
1
vote
3answers
10k views

How to Edit or Remove robots.txt on a WordPress powered website

When using Google's fetch test on my WordPress site, it reports that my robots.txt is blocking pages and resources from being indexed. Google reports that my Robots.txt looks like this: User-agent: ...
2
votes
2answers
509 views

Wildcard matches in robots.txt that allow crawling of all JS and CSS are not working

To allow the robots to crawl all CSS and JS files we have used the following code: Allow: /*.css$ Allow: /*.js$ But, some CSS and JS are still being blocked as stated when using the mobile friendly ...
1
vote
2answers
48 views

Blocking every URL with “?” in it

Will below Disallow function will block every URL which has "?" in it? Disallow: /*?*PrintPage=yes* I was actually checking few pages with "?" in them and they were not indexed in Google and then I ...
1
vote
0answers
31 views

SERP's saying meta data is blocked by the robots.txt file after issue resolved [closed]

Google crawled the robots.txt file when there was a mistake in it and the homepage was blocked. In the Google SERP's it says that the meta data is blocked by the robots.txt. The mistake in the file ...
3
votes
1answer
1k views

Do I need to add subdomain sitemap / robots.txt separately?

Say I have a domain example.com And it has a subdomain sub.example.com And another subdomain internal.example.com I have some links on example.com and sub.example.com for Google to crawl. But I don'...
1
vote
1answer
76 views

Robots.txt blocking homepage on mobile SERP's but not desktop

The meta data for the homepage on mobile is stating that the robots.txt file is blocking the page from being crawled. This is not the case for desktop. Desktop and mobile have the same domain, we ...
0
votes
1answer
485 views

Stopping Google from crawling my static domain

I use a cookieless sub domain static.example.com to serve all images, js, and css files. This static sub domain has as its root directory the same directory as the parent http://example.com. So static....
2
votes
2answers
913 views

Can I list multiple sitemaps inside one sitemap.xml and specify that in my robots.txt?

The sitemap engine I'm using generates a strange sitemap.xml. I need to add blog and shop sitemaps for those URLs too. Now in robots.txt I specify this path https://example.com/sitemap-shop.xml, but ...
2
votes
1answer
352 views

Help with robots meta tag and X-Robots-Tag

I have two questions about robots meta tag and X-Robots-Tag: If a page has both X-Robots-Tag HTTP header and robots meta tag, can this cause problems? For example: On HTTP response headers: X-...
2
votes
1answer
79 views

Blocking vs noindex to reduce crawl requests

I observed that GoogleBot is making a lot of duplicate requests for the same URLs from my website within a week. Amongst these requests a majority were for low/thin value pages(no or very low SERP,not ...
3
votes
2answers
315 views

Block third domain from being indexed

We have a web application that handles many websites that have test and production environment. This application has only one folder and it's hosted with IIS. We already have noindex nofollow, index ...
1
vote
2answers
111 views

Allow bots to read dynamically injected content

I've got a pretty large Angular SPA, and I currently use ?_escaped_fragment_ to serve up static versions of all our pages. I've discovered, however, that this often has issues with newly deployed/...
1
vote
1answer
53 views

Effect of incomplete Disallow rule in robots.txt file

Solved: Pages were being blocked by meta robots deliberately A lot of pages are being blocked in the robots.txt file and when I checked the file there were no indications of a rule that blocks these. ...
15
votes
4answers
6k views

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing? http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q My website was hacked which is now ...
1
vote
0answers
45 views

Not able to Open Website from Google Search [closed]

My website http://www.softnice.com (WARNING: Possibly hacked site) If I search in Google for "softnice" or "www.sofnice.com" I am not able to open my website. It shows: This www.softnice.com page ...
1
vote
1answer
144 views

Query String Pages - Do You Block Them via Robots.txt? Or Canonicals, `rel=“prev”/rel=“next”` do the Job

can someone please give me an advice on how to deal with query string pages like /index?page=323, ?min=0&max=217, ?dir=asc&order=name, ?mode=listand more like these. 2 known solutions for ...
3
votes
2answers
135 views

Google claims not to find my robots.txt

Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it. But there is a robots.txt file (https://soundhaus-band....
2
votes
1answer
50 views

Google Search Console error with path to sitemap.xml

I have site-shop and the platform created a sitemap.xml automatically. But in Google Search Console there is an error because robots.txt like this: User-agent: * Disallow: /my/ Disallow: /cart/ ...
2
votes
1answer
81 views

Hiding search pages via robots.txt

I have full text search engine installed on my site, accessible from URL like /search/<search-query>, and search landing in /search, containing large "Search" button and some popular tags. Do I ...
5
votes
3answers
1k views

Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources

I have a robots.txt file: User-agent:* Disallow:/path/page Disallow:/path/ Allow:/ The disallowed path is still getting crawled. I have searched this problem and what they said, order of precedence ...
4
votes
2answers
470 views

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google? Yesterday I ...
3
votes
1answer
389 views

Google Webmasters Blocked Resources/Robots.txt File?

For some reason Google Webmaster Tools insists that pretty much all js and css resources are blocked on every page of my website and when I try to render my pages in GWT they will only partially ...
2
votes
1answer
35 views

Allow crawlers to pass link juice from similar and duplicated user created pages

My users commonly create new pages with similar/duplicate content. For example: http://example.com/post?123 http://example.com/post?456 I've therefore disallowed these pages in the robots.txt file. ...
1
vote
1answer
147 views

Using asteriks in robots.txt

Can spiders crawl the sites if there are no asterisks used on the robots.txt? For example: User-agent: * Disallow: .*/ping/.* then User-agent: * Disallow: /ping/ Please can someone confirm this, ...
1
vote
3answers
155 views

Applebot not crawling sitemap.xml

On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages. We have no robots.txt restrictions for that bot, on the contrary we ...
1
vote
1answer
75 views

Will Google remove links restricted by the robots.txt from the soft 404 list?

I have updated a shop system to a newer version. I now get huge number of soft 404 errors in the Webmaster Tools because of links that have been indexed in the old version of that shop. I could lock ...
3
votes
1answer
1k views

How do I edit robots.txt on a Google site?

I need to make changes to someone's robots.txt file, but their site is managed by Google (so no FTPing). I have full access to the site via the browser (Site Actions / Manage Site, etc.) but how do I ...
1
vote
1answer
120 views

Index site via domain name, but not via direct IP

For some context, I'm modifying somebody's website. This particular site is not yet published - it is accessible via direct IP. Problem: Google has crawled the direct IP, and people are reaching the ...
3
votes
2answers
190 views

My robots.txt is showing on one line after uploading

So I have a weird issue, and I'm not sure if its caused due to my error or what, but when I upload a robots.txt like below: User-Agent: * Disallow: /dev/ Disallow:/wp-admin/ Disallow:/wp-includes/ ...
9
votes
3answers
16k views

How can I use robots.txt to disallow subdomain only?

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and ...
1
vote
1answer
50 views

What is the correct way to write my “robots.txt” file? [closed]

I have written the following code inside my robots.txt file: User-agent: * Disallow: User-agent: googlebot Allow: / Is my robots.txt is correct?
1
vote
1answer
243 views

Unexplainable non-existing page requested for months

While I am quite familiar with bots requesting well-know pages as used for WordPress and phpMyAdmin on any site the can reach, I'm now dealing with a bot requested a page I really can't explain. For ...
1
vote
1answer
57 views

Bots cannot crawl old parts of site with “Disallow: /old/” and “Disallow: /old”?

Correct me if I am wrong, this means bots cannot crawl the old parts of the site? User-agent: * Disallow: /old/ Disallow: /old Since it has been days that I have not received any traffic and found ...
2
votes
2answers
531 views

Prevent Google from seeing “svg” as a content keyword without using .htaccess

Is there a simple way to do this without using "X-Robots-Tag: noarchive" and robots.txt ? Because in some shared hosting packages it is not possible to modify this kind of values. I have came across ...
0
votes
1answer
293 views

Google-Image bot not respecting robots.txt

I have the following in my robots.txt file: # Block Googlebot-Image from cache User-agent: Googlebot-Image Disallow: /media/catalog/product/cache/ # Crawlers Setup User-agent: * # Directories ...
3
votes
4answers
119 views

Proper robots.txt block URL

I want to block this URL: /services/online.html?dir=asc&limit=12&order=price but I want to allow this one: /services/online.html. I have added this on the robots.txt: Allow: /services/online....
5
votes
3answers
461 views

Remove large number of useless page from Google index using robots.txt

A large number of auto-generated pages from the search of my site has been indexed by Google due to an error. I'm trying to clean this in order to be sure that only the most high quality content is ...
0
votes
3answers
429 views

Can backlinks to robots.txt blocked pages boost domain's ranking?

So, imagine, we were blocking some URLs via robots.txt, but apparently these pages were getting backlinks. Later, we decided to redirect these blocked pages to another domain of ours. ~1.5 month ...
1
vote
1answer
154 views

Google not indexing site, robots being blocked

I have a small static website which allows customers to purchase a PDF containing some data based from an input from them. The PDF's are a page or two and follow a standard template, and various ...
6
votes
3answers
21k views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
0
votes
2answers
527 views

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt. I have the following sole entry for my /robots.txt: location = /robots.txt { return 200 "## $host ## \n #Dear ...
3
votes
1answer
467 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
0
votes
0answers
292 views

Googlebot is unable to access my site's robots.txt

I am having this issue: "Google couldn't crawl your site because we were unable to access your site's robots.txt file." I have already tried all the suggestions provided in the related problems but ...
2
votes
1answer
342 views

Include “www” in sitemap URL in robots.txt when site redirects to no-www?

Website is redirected to remove the www so www.example.com is redirecting to example.com. Question is do I need to include the www in the sitemap for the robots.txt?
2
votes
0answers
168 views

Google search console reports URLs blocked by robots.txt even though they shouldn't be

Recently I sent a sitemap to Google Search Console and I got some warnings saying that some urls are blocked by the robots.txt. I checked it and there is no single line that blocks the corresponding ...