Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
0
votes
1answer
7k views

How block only Yandex bot

Can you show me how will look robots.txt when I block only yandex bot, allow Google bot and block Yandex bot.
1
vote
2answers
403 views

How does robots.txt work with sites in subfolders?

I have a single web host with a number of other parked domains/sites in sub-directories, like this: example.com is the primary site and root directory of the web hosting. example.com/www.example2....
1
vote
2answers
183 views

Is rel=“nofollow” necessary for a sponsered link through a redirect script disallowed by robots.txt?

Paid links (e.g. affiliate links) should have the rel="nofollow" attribute added them to prevent passing on PageRank. But is this necessary if the URL is already blocked by robots.txt? Example: Let's ...
2
votes
0answers
229 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
1
vote
1answer
1k views

Use robots.txt to prevent privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google

I need to block pages such as privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google. They might be unoriginal. I use them for my multiple websites and they ...
1
vote
0answers
33 views

Open ports on a website [closed]

Alright. I risk sounding like an absolute newbie but I've got to know this. A friend and I paid a developer to develop a website for a start-up we had come up with. The idea fell through and the ...
0
votes
2answers
107 views

What's wrong with my robots.txt file to block Google?

I have a website and I don't want this website to be visible on Google. So I deployed a robots.txt file. But the homepage of the website is still visible on Google. Here is the full code of my robots....
2
votes
3answers
1k views

Google not crawling my site (robots.txt error)

I'm currently performing SEO for my client's project. I'm kind of new to this, so please bear with me. I've read a lot mixed reviews on inclusion of robots.txt (some say it's good to include even if ...
4
votes
2answers
11k views

Changing sitemap.xml and robots.txt after moving from http to https

I am migrating a website from http to https entirely, all http urls will have 301 redirects to their https counterparts. From https://support.google.com/webmasters/answer/6033049 We reference our ...
4
votes
1answer
2k views

How to have 2 sitemaps - 1 for main site the other for wordpress in subfolder

Background My site runs on a dedicated server and and has Wordpress in a subfolder. I have my mainsite which is not wordpress and that has a sitemap - for example example.com/sitemap.xml ... that ...
2
votes
2answers
125 views

Google Search Console: Blocked by robots (Screenshot)

I just discovered that there was a huge decrease in the number of pages that are blocked by robots in my search console. It is an online shop so i blocked many product pages and filters to avoid ...
1
vote
0answers
10 views

Geographical landing pages not showing up on Google search results [duplicate]

Problem we have: One markets' geographical landing pages do not appear on search results. Few months ago we discovered that robots.txt are blocking Google from indexing that markets' pages. We fixed ...
4
votes
2answers
3k views

Prevent indexing of image or file

I understand that you can do the following to generally prevent images or pages from getting indexed. Add to the page's meta section: <meta name="robots" content="noindex" /> Edit the robots....
2
votes
1answer
213 views

robots.txt proper way to set it up

If you want to allow all search engines to crawl a site and only block one specific folder, is this correct? User-agent: * Disallow: /folder_name/ If you don't want to block any folder then is this ...
2
votes
1answer
638 views

“Is robots.txt blocking important pages?” in Search Console regarding static folder

This is the contents of my robots.txt file: User-agent: * Disallow: /static/* Disallow: /templates/* Disallow: /translations/* Sitemap: https://xx.xxx.com/sitemap.xml However I don't know why I am ...
2
votes
1answer
115 views

robots.txt for website hosted in a subdirectory

I have 2 website which is hosted in a shared hosting. 1st website example.com hosted in a root directory as /public_html/ 2nd website example2.com hosted in a sub-directory directory as /public_html/...
0
votes
2answers
188 views

Spider-Trap on a GitHub Site

I have a GitHub site and I hate web-crawlers that disobey or ignore robots.txt. How would I set up a Spider-Trap on a GitHub site that the robots.txt disallows the trap, so the good bots aren't ...
1
vote
3answers
11k views

How to Edit or Remove robots.txt on a WordPress powered website

When using Google's fetch test on my WordPress site, it reports that my robots.txt is blocking pages and resources from being indexed. Google reports that my Robots.txt looks like this: User-agent: ...
2
votes
2answers
555 views

Wildcard matches in robots.txt that allow crawling of all JS and CSS are not working

To allow the robots to crawl all CSS and JS files we have used the following code: Allow: /*.css$ Allow: /*.js$ But, some CSS and JS are still being blocked as stated when using the mobile friendly ...
1
vote
2answers
48 views

Blocking every URL with “?” in it

Will below Disallow function will block every URL which has "?" in it? Disallow: /*?*PrintPage=yes* I was actually checking few pages with "?" in them and they were not indexed in Google and then I ...
1
vote
0answers
32 views

SERP's saying meta data is blocked by the robots.txt file after issue resolved [closed]

Google crawled the robots.txt file when there was a mistake in it and the homepage was blocked. In the Google SERP's it says that the meta data is blocked by the robots.txt. The mistake in the file ...
3
votes
1answer
1k views

Do I need to add subdomain sitemap / robots.txt separately?

Say I have a domain example.com And it has a subdomain sub.example.com And another subdomain internal.example.com I have some links on example.com and sub.example.com for Google to crawl. But I don'...
1
vote
1answer
76 views

Robots.txt blocking homepage on mobile SERP's but not desktop

The meta data for the homepage on mobile is stating that the robots.txt file is blocking the page from being crawled. This is not the case for desktop. Desktop and mobile have the same domain, we ...
0
votes
1answer
524 views

Stopping Google from crawling my static domain

I use a cookieless sub domain static.example.com to serve all images, js, and css files. This static sub domain has as its root directory the same directory as the parent http://example.com. So static....
2
votes
2answers
984 views

Can I list multiple sitemaps inside one sitemap.xml and specify that in my robots.txt?

The sitemap engine I'm using generates a strange sitemap.xml. I need to add blog and shop sitemaps for those URLs too. Now in robots.txt I specify this path https://example.com/sitemap-shop.xml, but ...
2
votes
1answer
358 views

Help with robots meta tag and X-Robots-Tag

I have two questions about robots meta tag and X-Robots-Tag: If a page has both X-Robots-Tag HTTP header and robots meta tag, can this cause problems? For example: On HTTP response headers: X-...
2
votes
1answer
79 views

Blocking vs noindex to reduce crawl requests

I observed that GoogleBot is making a lot of duplicate requests for the same URLs from my website within a week. Amongst these requests a majority were for low/thin value pages(no or very low SERP,not ...
3
votes
2answers
356 views

Block third domain from being indexed

We have a web application that handles many websites that have test and production environment. This application has only one folder and it's hosted with IIS. We already have noindex nofollow, index ...
1
vote
2answers
112 views

Allow bots to read dynamically injected content

I've got a pretty large Angular SPA, and I currently use ?_escaped_fragment_ to serve up static versions of all our pages. I've discovered, however, that this often has issues with newly deployed/...
1
vote
1answer
58 views

Effect of incomplete Disallow rule in robots.txt file

Solved: Pages were being blocked by meta robots deliberately A lot of pages are being blocked in the robots.txt file and when I checked the file there were no indications of a rule that blocks these. ...
15
votes
4answers
6k views

Site was hacked, need to remove all URLs starting with + from Google, use robots.txt?

Could you please let me know how to block such URLs from robots.txt for Googlebots to stop indexing? http://www.example.com/+rt6s4ayv1e/d112587/ia0g64491218q My website was hacked which is now ...
1
vote
0answers
45 views

Not able to Open Website from Google Search [closed]

My website http://www.softnice.com (WARNING: Possibly hacked site) If I search in Google for "softnice" or "www.sofnice.com" I am not able to open my website. It shows: This www.softnice.com page ...
1
vote
1answer
157 views

Query String Pages - Do You Block Them via Robots.txt? Or Canonicals, `rel=“prev”/rel=“next”` do the Job

can someone please give me an advice on how to deal with query string pages like /index?page=323, ?min=0&max=217, ?dir=asc&order=name, ?mode=listand more like these. 2 known solutions for ...
3
votes
2answers
138 views

Google claims not to find my robots.txt

Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it. But there is a robots.txt file (https://soundhaus-band....
2
votes
1answer
51 views

Google Search Console error with path to sitemap.xml

I have site-shop and the platform created a sitemap.xml automatically. But in Google Search Console there is an error because robots.txt like this: User-agent: * Disallow: /my/ Disallow: /cart/ ...
2
votes
1answer
93 views

Hiding search pages via robots.txt

I have full text search engine installed on my site, accessible from URL like /search/<search-query>, and search landing in /search, containing large "Search" button and some popular tags. Do I ...
5
votes
3answers
1k views

Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources

I have a robots.txt file: User-agent:* Disallow:/path/page Disallow:/path/ Allow:/ The disallowed path is still getting crawled. I have searched this problem and what they said, order of precedence ...
4
votes
2answers
486 views

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google? Yesterday I ...
3
votes
1answer
399 views

Google Webmasters Blocked Resources/Robots.txt File?

For some reason Google Webmaster Tools insists that pretty much all js and css resources are blocked on every page of my website and when I try to render my pages in GWT they will only partially ...
2
votes
1answer
36 views

Allow crawlers to pass link juice from similar and duplicated user created pages

My users commonly create new pages with similar/duplicate content. For example: http://example.com/post?123 http://example.com/post?456 I've therefore disallowed these pages in the robots.txt file. ...
1
vote
1answer
160 views

Using asteriks in robots.txt

Can spiders crawl the sites if there are no asterisks used on the robots.txt? For example: User-agent: * Disallow: .*/ping/.* then User-agent: * Disallow: /ping/ Please can someone confirm this, ...
1
vote
3answers
160 views

Applebot not crawling sitemap.xml

On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages. We have no robots.txt restrictions for that bot, on the contrary we ...
1
vote
1answer
77 views

Will Google remove links restricted by the robots.txt from the soft 404 list?

I have updated a shop system to a newer version. I now get huge number of soft 404 errors in the Webmaster Tools because of links that have been indexed in the old version of that shop. I could lock ...
3
votes
1answer
1k views

How do I edit robots.txt on a Google site?

I need to make changes to someone's robots.txt file, but their site is managed by Google (so no FTPing). I have full access to the site via the browser (Site Actions / Manage Site, etc.) but how do I ...
1
vote
1answer
124 views

Index site via domain name, but not via direct IP

For some context, I'm modifying somebody's website. This particular site is not yet published - it is accessible via direct IP. Problem: Google has crawled the direct IP, and people are reaching the ...
3
votes
2answers
207 views

My robots.txt is showing on one line after uploading

So I have a weird issue, and I'm not sure if its caused due to my error or what, but when I upload a robots.txt like below: User-Agent: * Disallow: /dev/ Disallow:/wp-admin/ Disallow:/wp-includes/ ...
10
votes
3answers
18k views

How can I use robots.txt to disallow subdomain only?

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and ...
1
vote
1answer
51 views

What is the correct way to write my “robots.txt” file? [closed]

I have written the following code inside my robots.txt file: User-agent: * Disallow: User-agent: googlebot Allow: / Is my robots.txt is correct?
1
vote
1answer
250 views

Unexplainable non-existing page requested for months

While I am quite familiar with bots requesting well-know pages as used for WordPress and phpMyAdmin on any site the can reach, I'm now dealing with a bot requested a page I really can't explain. For ...
1
vote
1answer
57 views

Bots cannot crawl old parts of site with “Disallow: /old/” and “Disallow: /old”?

Correct me if I am wrong, this means bots cannot crawl the old parts of the site? User-agent: * Disallow: /old/ Disallow: /old Since it has been days that I have not received any traffic and found ...

1 2 3 4 5 13