Questions tagged [robots.txt]

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

Filter by
Sorted by
Tagged with
1
vote
1answer
230 views

Is sitemap in sitemap_index getting indexed by Google webmaster?

it sounds a silly question, but I actually could not find a reference to it. I have created a sitemap_index with about 100-ish sitemaps. No errors. I submitted only the sitemap_index to Google ...
1
vote
1answer
37 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: /...
3
votes
1answer
3k views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
0answers
683 views

Google can't crawl my rewritten URLs

I tried to submit my robots.txt and sitemap.xml for hours now - changing directories and .htacces rules like a horse. For some stupid reason Google can't crawl my pages and gives me warnings, that my ...
2
votes
0answers
268 views

How to make nginx override web app route for sitemap and robots?

I would like to make nginx serve sitemap and robots. My problem is that route http://example.com/ is handled by flask, configured to handle request with certain patterns - http://example.com/sitemap....
3
votes
1answer
478 views

Can HTML sitemaps be listed in robots.txt just like XML sitemaps?

I will have two sitemaps for my website. The first sitemap is the .xml version for the search engines. The second sitemap is a sitemap.php on my website for the users. I know I can list the sitemap....
3
votes
1answer
103 views

After changing domain name Google's index says pages are blocked by robots.txt despite Allow: /

Two weeks ago I changed the address of my site. I forgot to renew my domain, and for a few days my site was unreachable. This resulted in google indexing my site improperly. There is a robot.txt error ...
2
votes
2answers
60 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
3
votes
2answers
524 views

Google can't access non-existent robots.txt

I set up a website a few weeks ago and I'm trying to get Google to crawl it. When logging into Google's Search Console (Webmaster Tools) and within: Crawl > Crawl Errors It reports: Google ...
1
vote
2answers
437 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
2answers
397 views

How to block subdomains under my main domain

I want all my subdomains to not be indexed under my domain, that means when I search in Google for example.com to not appear in index test.example.com, but when I search test.example.com to be indexed....
2
votes
2answers
706 views

Google Search Console: Severe health issues, some important page is blocked by robots.txt

I have just added a new property to the Google search console. I've added it as http://example.com and http://www.example.com. I receive the following error message: Severe health issues are found ...
1
vote
1answer
130 views

The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix?

We have a website: www.example.com which redirects to https://www.example.com. Therefore when you type www.example.com/robots.txt you are redirected to https://www.example.com/robots.txt. Another ...
2
votes
1answer
585 views

cookies and useragent is not enough ? to emulate browser?

I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt Let's take an example, I need to get this page: http://www.aliexpress.com/...
1
vote
1answer
64 views

Should I block a dynamically generated data results page in robots.txt?

I have a tiny two page site: the parameters page and results page. The results page can either work with URL parameters or hidden POST ones. The output is a giant table of data that contains rankings ...
2
votes
0answers
100 views

Google Webmaster Tools reports URLs blocked by robots.txt on my AVS tube website

I added a tube (with AVS script) in Google Webmaster Tools, but the issue is with the robots.txt file which is blocking everything. Here is my AVS tube robots.txt: User-agent: * Allow: / Disallow: /...
2
votes
1answer
66 views

Google indexed work-in-progress page. how to delete it from Google index?

I developed a page under the following URL: example.org/dev Since the development is finished, I moved the page to the root at example.org/ For some reason, Google indexed the page during ...
7
votes
5answers
1k views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
1
vote
1answer
716 views

Robots.txt disallow all except one directory and access to sitemap

robots.txt containts following instructions: User-agent: * Allow: /public/ Disallow: / Sitemap: http://sitedomain.com/sitemapindex.xml it seems that in this case sitemapindex won't be accessible for ...
0
votes
2answers
375 views

Sitemap XML title in search results

I have a website that uses an XML sitemap and a robots.txt file. If I do a Google search of: site: mywebsite.co.uk "sitemap" I see the usual Google response, similar to the following image, but ...
14
votes
2answers
8k views

Can robots.txt be in a server's sub-directory?

I have a sub-directory that I would like to hide from the search engine web crawlers. One way to do this is to use a robots.txt in the root directory of the server (the standard way). However, anyone ...
2
votes
2answers
3k views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
3
votes
1answer
199 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
4
votes
3answers
164 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
1
vote
1answer
34 views

Remove website pages from search results, but keep them crawled

I would like to remove all pages of my website from the search results, except the home page. Is there a way to do this and keep the pages crawlable? Or is there no way except robots.txt?
3
votes
1answer
382 views

How do I disallow variable parent directories and subdirectory with robots.txt?

I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages: /category/name1/page/1 /category/name1/page/2 /category/etc/page/3 But I want to allow ...
1
vote
1answer
591 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
5
votes
4answers
3k views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it is ...
0
votes
0answers
21 views

Robots.txt Query String - Couldn't Disallow [duplicate]

I have url : http://www.website.com/shopping/books/?b=9 I disallowed /?b=9 from robots.txt as User-agent: * Disallow: /?b=9 But when i test this from Google Webmaster, on Robots.txt Test Tools, it ...
2
votes
2answers
141 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
2
votes
1answer
46 views

Using canonical links to eliminate “similar” search results in google

In our site, we have links as such: /{{category-name}}.html - will display all products /category-name/filters/{{some-filter}}.html - will display filtered products The problem we are facing is ...
0
votes
0answers
11 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
559 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
2
votes
1answer
245 views

Google reports soft 404 errors and doesn't remove several of my website URLs from the index after I 301 redirect them

I want to remove multiple URL in my website (developed in ASP MVC). All these URL have the format www.example.com/planning/xxx I have already changed the code to make these URLs 301 redirect. I ...
1
vote
0answers
30 views

Why does my staging site show up in Google even though my robots.txt file shouldn't allow it? [duplicate]

I'm having a lot of trouble with something that should be simple. I don't want my development site indexed by Google. According to Google's instructions, and just about everything else I can find says ...
3
votes
2answers
358 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: www....
3
votes
1answer
85 views

Point Domain Names With keyword to main website?

This is Question has been asked several time on Stack Exchange. I have read all those answer, still there are certain some technical specification answered. After lot of research on Internet , I ...
4
votes
3answers
73 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
1
vote
2answers
603 views

Proper content for robots.txt on server maintenance page?

I know that the proper server response header should be 503 Service Unavailable, to let the search bots know that they shouldn't index the pages like Update in progress and Server maintenance. But ...
3
votes
1answer
353 views

Sitemap in different domain and robots.txt crawling

I have the following subdomains: www.example.com api.example.com The subdomain api.example.com is supposed to be invisible to Google. Its robots.txt had a Disallow / directive. But I don't want any ...
2
votes
1answer
496 views

How much time does robots.txt remain in Google cache?

The old robots.txt alerts me with warning about locked URL. With the new one, the URL passes the tester in Google Webmaster. But when I resend the sitemap I get the same error!
0
votes
2answers
1k views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
0
votes
0answers
84 views

Google reports that robots.txt is blocking site - but I don't think it is

I have a site map at http://www.example.com/sitemap_index.xml that has been sumbitted to Google Webmaster Tools, however... Google is reporting that my sitemap is being blocked by robots.txt. My ...
10
votes
1answer
1k views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
0
votes
1answer
81 views

Is a standard WordPress robots.txt preventing indexing of this site?

Our website http://www.example.com has a sub-domain http://jobs.example.com that uses a tool which shares our job listings. We would like for job listings to become indexed in Google search results. ...
1
vote
0answers
516 views

Magento Robots.txt and blocking cached images

We have a Magento based store. Images are resized and served cached dynamically. When the image cache in Magento is cleared, there are generated new paths and files. And I get 404 errors for them ...
4
votes
2answers
78 views

Should I block an .htm resource in robots.txt

We use scripts to create static .htm files to show content on our site. Mainly in an effort to reduce database calls. For example, a script that builds a file every 5 minutes that contains a list of ...
2
votes
2answers
650 views

How do robots respond to no user-agent in robots.txt?

I'm just curious because I can't seem to find any mention of it anywhere. What is the standard (or at the very least typical) response to robots.txt without a user-agent parameter? Does it assume the ...
1
vote
2answers
265 views

can't remove blocks on my resources in google webmaster tools

I received an alert from Google Webmaster Tools a couple weeks ago that stated: "Google systems have recently detected an issue with your homepage that affects how well our algorithms render and ...
0
votes
1answer
437 views

URLs with parameters & canonical tags. Block them via robots.txt vs GSC URL parameters?

I have some URLs with parameters, the parameters aren't of the filtering kind, they don't create millions of duplicates. The parameters are simple being used for tracking purpose, content doesn't ...