Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
1
vote
0answers
9 views

Wordpress Website not indexing in google, no error on google webmaster [duplicate]

I just recognized my SEO Score is 0 and that my wordpress website is not indexed on google search engine, no automatic filling as well. I am not an expert, I never used a sitemap, nor htaccess but it ...
2
votes
2answers
58 views

Moving a site from one subdomain to another subdomain, what to do with pages which aren't “mappable”?

In the process of moving a site from one subdomain to another subdomain. For a lot of the subfolders a 301 redirect to their respective pages on the new site. However what should I do with subdomains ...
2
votes
1answer
68 views

Parts of my site are not getting indexed after changing the URLs, adding redirects, and blocking the old URLs in robots.txt

I have a website with some URLs and later I optimized the URLs and changed the link structure. I blocked the old URLs with robots.txt and suggested redirections from old to new URLs, and then did ...
1
vote
1answer
266 views

Prevent indexing of site search results

Users have the ability to search on my site. This function renders a search results page that has occasionally been indexed by google and served in SERPs. Normally this is not an issue, but when ...
1
vote
1answer
458 views

Internal Search Results: NOINDEX or robots.txt Blocking?

We have our own internal search results pages for our download section. These results are currently crawlable but with NOINDEX meta tags so they don't appear in the search index. People link to ...
0
votes
1answer
875 views

How to implement HTTPS redirection for static resources such as robots.txt in legacy ASP.NET website?

We switched to HTTPS in our legacy ASP.NET site about a month ago. We had already implemented redirection with HTTP status code 301 'Moved permanently' for outdated aspx resources in the app's Global....
8
votes
3answers
2k views

When you move a site via a 301 redirect should you setup a robots.txt disallowing robots to crawl the old address?

A site I am working on moved a subdomain to another subdomain via a 301 redirect. However when checking robots.txt of the old subdomain, it has a robots.txt which disallows search engine web crawlers ...
1
vote
1answer
1k views

'Sitemap contains urls which are blocked by robots.txt.' Warning - However robots.txt file doesn't appear to be blocking anything

Our site is built on WordPress. Whilst in the development stage, we had the option to Discourage search engines from indexing this site checked (Settings Reading). We have now made the site live and ...
2
votes
1answer
112 views

Should I prevent search engines from indexing empty user profile pages? If so, how much content is enough for indexing?

I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available: The main user page (about me, last activity, ...) Several book ...
2
votes
2answers
165 views

Protocol Agnostic Robots Sitemap

Recently, I have enabled all my servers to serve everything over HTTP and HTTPS. Users can access any site via http://www.example.com or https://www.example.com. All pages are identical between the ...
2
votes
2answers
508 views

How to tell google that domain no longer exists but still make redirects from that domain

I have two domains: example.com and dev.example.net. example.com is the active one and dev.example.net should be invisible to Google. dev.example.net has 301 redirects to example.com and it has its ...
2
votes
1answer
69 views

Deleting a URL in Google Search Console causes a warning that something important has been blocked

I have removed a page on my website and put the URL as Disallow in robots.txt and I removed the URL in Google Search Console (Webmaster Tools). But now Google is telling me that I removed a very ...
1
vote
2answers
66 views

Will it effect SEO if there are 30 links on a page where the destination is blocked by robots.txt

I have a webpage on which there are around 30 data points which link to 30 different pages on my own site which are non-crawlable and have been blocked via robots.txt My question is if we use href ...
1
vote
1answer
38 views

Removing content in specific directories from Google search results

We are using a plugin on our website to handle job application submissions. I found out recently that the applications were being uploaded into a public directory. We've had some individuals approach ...
5
votes
2answers
2k views

Redirecting robots.txt when moving to a new domain

I'm moving a site to a new domain, to a local tld. I already redirected everything successfully to the new domain, except the robots.txt. My question would be that is it needed to redirect the ...
2
votes
1answer
40 views

Huge tables won't fit on mobile, can I tell Google the page is desktop only?

I'm working on a product comparison page that is composed of a giant table listing the attributes of a great many products. It's useful on a desktop for those who want to dig into the details of ...
1
vote
1answer
2k views

“Crawl-delay” rule ignored by googlebot

In Google Webmaster Tools this warning showing for site: The yellow signal indicating "rule ignored by googlebot". How can I fix those warning?
1
vote
1answer
2k views

Firebase hosting, sitemaps and robots.txt

i tried searching on internet about this question and how Firebase handle indexing and how owner of the content running on Firebase hosting can influence crawling by Google bot. Personally, i am ...
1
vote
1answer
150 views

How to disallow bulk urls from robots.txt?

I built a dynamic sitemap generator on my platform. By mistake I generated some 300+ wrong url along with the right ones. And they have been there for couple weeks now. Recently I found out this ...
2
votes
1answer
71 views

Why wouldn't a website apear in Google search results after robots.txt update to clean up hacked site?

I have a WordPress website which was hacked couple of days ago. I have tried to add up some security. By following couple of tutorials, I have updated my robots.txt file. But now my website is not ...
4
votes
1answer
519 views

Are user agent names case-sensitive in robots.txt?

I'm blocking various bots in robots.txt and I was wondering if their names are case-sensitive. For example: User-agent: grapeshot Disallow: / Would it matter if I use grapeshot or GrapeShot?
3
votes
1answer
587 views

HTTP To HTTPS sitemap issues

Today I moved my site from HTTP to HTTPs successfully. Code I used: if($_SERVER["HTTPS"] != "on"){ if(php_sapi_name() === 'cli'){ } else{ $host = $_SERVER['HTTP_HOST']; ...
1
vote
1answer
32 views

How to serve a corresponding robots.txt file for each website in the same directory?

I couldn't find the exact answer for this particular issue. I have two add-on domains https://domain001.com and https://domain002.com hosted in /public_html/domain001/ and public_html/domain002/ A ...
0
votes
1answer
67 views

Fetch as Google, ad network has robots.txt with Disallow all, is it cloacking?

we run ads on our site and I've noticed that at the eyes of google the page isn't the same as at those of the users'. In other words, ads are not being displayed because the Ad network we use has ...
2
votes
1answer
25 views

Should article preview pages be crawled and indexed by search engines?

I have a page called "all articles" that loads previews of articles using AJAX. Because of that, the content won't be visible to search engines, so the page would be considered "poor content". Since ...
2
votes
0answers
248 views

Google Blocked Resources showing incorrect data

In Google Webmaster Blocked Resources tab it is showing 500+ blocked resource (like .js and .css), but when I tested those URL in robots.txt Tester it is showing Allowed, then why not Google bot is ...
1
vote
1answer
271 views

To allow crawling of all but a specific folder, do I need to include an empty disallow directive in robots.txt?

If I want my website to be crawled, do I need an "empty" disallow? Is there any difference between User-agent: * Disallow: Disallow: /folder/ and User-agent: * Disallow: /folder/ I have seen a ...
0
votes
1answer
5k views

How block only Yandex bot

Can you show me how will look robots.txt when I block only yandex bot, allow Google bot and block Yandex bot.
1
vote
2answers
358 views

How does robots.txt work with sites in subfolders?

I have a single web host with a number of other parked domains/sites in sub-directories, like this: example.com is the primary site and root directory of the web hosting. example.com/www.example2....
1
vote
2answers
162 views

Is rel=“nofollow” necessary for a sponsered link through a redirect script disallowed by robots.txt?

Paid links (e.g. affiliate links) should have the rel="nofollow" attribute added them to prevent passing on PageRank. But is this necessary if the URL is already blocked by robots.txt? Example: Let's ...
2
votes
0answers
203 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
1
vote
1answer
1k views

Use robots.txt to prevent privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google

I need to block pages such as privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google. They might be unoriginal. I use them for my multiple websites and they ...
1
vote
0answers
33 views

Open ports on a website [closed]

Alright. I risk sounding like an absolute newbie but I've got to know this. A friend and I paid a developer to develop a website for a start-up we had come up with. The idea fell through and the ...
0
votes
2answers
102 views

What's wrong with my robots.txt file to block Google?

I have a website and I don't want this website to be visible on Google. So I deployed a robots.txt file. But the homepage of the website is still visible on Google. Here is the full code of my robots....
2
votes
3answers
1k views

Google not crawling my site (robots.txt error)

I'm currently performing SEO for my client's project. I'm kind of new to this, so please bear with me. I've read a lot mixed reviews on inclusion of robots.txt (some say it's good to include even if ...
3
votes
2answers
9k views

Changing sitemap.xml and robots.txt after moving from http to https

I am migrating a website from http to https entirely, all http urls will have 301 redirects to their https counterparts. From https://support.google.com/webmasters/answer/6033049 We reference our ...
4
votes
1answer
1k views

How to have 2 sitemaps - 1 for main site the other for wordpress in subfolder

Background My site runs on a dedicated server and and has Wordpress in a subfolder. I have my mainsite which is not wordpress and that has a sitemap - for example example.com/sitemap.xml ... that ...
2
votes
2answers
115 views

Google Search Console: Blocked by robots (Screenshot)

I just discovered that there was a huge decrease in the number of pages that are blocked by robots in my search console. It is an online shop so i blocked many product pages and filters to avoid ...
1
vote
0answers
10 views

Geographical landing pages not showing up on Google search results [duplicate]

Problem we have: One markets' geographical landing pages do not appear on search results. Few months ago we discovered that robots.txt are blocking Google from indexing that markets' pages. We fixed ...
4
votes
2answers
3k views

Prevent indexing of image or file

I understand that you can do the following to generally prevent images or pages from getting indexed. Add to the page's meta section: <meta name="robots" content="noindex" /> Edit the robots....
2
votes
1answer
132 views

robots.txt proper way to set it up

If you want to allow all search engines to crawl a site and only block one specific folder, is this correct? User-agent: * Disallow: /folder_name/ If you don't want to block any folder then is this ...
2
votes
1answer
630 views

“Is robots.txt blocking important pages?” in Search Console regarding static folder

This is the contents of my robots.txt file: User-agent: * Disallow: /static/* Disallow: /templates/* Disallow: /translations/* Sitemap: https://xx.xxx.com/sitemap.xml However I don't know why I am ...
2
votes
1answer
95 views

robots.txt for website hosted in a subdirectory

I have 2 website which is hosted in a shared hosting. 1st website example.com hosted in a root directory as /public_html/ 2nd website example2.com hosted in a sub-directory directory as /public_html/...
0
votes
2answers
177 views

Spider-Trap on a GitHub Site

I have a GitHub site and I hate web-crawlers that disobey or ignore robots.txt. How would I set up a Spider-Trap on a GitHub site that the robots.txt disallows the trap, so the good bots aren't ...
1
vote
3answers
9k views

How to Edit or Remove robots.txt on a WordPress powered website

When using Google's fetch test on my WordPress site, it reports that my robots.txt is blocking pages and resources from being indexed. Google reports that my Robots.txt looks like this: User-agent: ...
2
votes
2answers
466 views

Wildcard matches in robots.txt that allow crawling of all JS and CSS are not working

To allow the robots to crawl all CSS and JS files we have used the following code: Allow: /*.css$ Allow: /*.js$ But, some CSS and JS are still being blocked as stated when using the mobile friendly ...
1
vote
2answers
48 views

Blocking every URL with “?” in it

Will below Disallow function will block every URL which has "?" in it? Disallow: /*?*PrintPage=yes* I was actually checking few pages with "?" in them and they were not indexed in Google and then I ...
1
vote
0answers
31 views

SERP's saying meta data is blocked by the robots.txt file after issue resolved [closed]

Google crawled the robots.txt file when there was a mistake in it and the homepage was blocked. In the Google SERP's it says that the meta data is blocked by the robots.txt. The mistake in the file ...
3
votes
1answer
944 views

Do I need to add subdomain sitemap / robots.txt separately?

Say I have a domain example.com And it has a subdomain sub.example.com And another subdomain internal.example.com I have some links on example.com and sub.example.com for Google to crawl. But I don'...
1
vote
1answer
75 views

Robots.txt blocking homepage on mobile SERP's but not desktop

The meta data for the homepage on mobile is stating that the robots.txt file is blocking the page from being crawled. This is not the case for desktop. Desktop and mobile have the same domain, we ...