robots.txt : allow some, disallow all the rest strategy

Question

My question is about how I should build robots.txt for my Web site.

Not so long ago, I used to only disallow a few pages like this:

User-agent: *
Disallow: /webapp_1/page_1_1_d.jsp
Disallow: /webapp_1/page_1_2_d.jsp
[...]
Disallow: /webapp_1/page_1_n1_d.jsp

Disallow: /webapp_2/page_2_1_d.jsp
Disallow: /webapp_2/page_2_2_d.jsp
[...]
Disallow: /webapp_2/page_2_n2_d.jsp

Recently, I decided instead, to make a list of the pages I wanted to allow and to disallow all the rest, like this:

User-agent: *
Allow: /sitemap.xml
Allow: /webapp_1/page_1_1.jsp
Allow: /webapp_1/page_1_2.jsp
[...]
Allow: /webapp_1/page_1_n3.jsp

Allow: /webapp_2/page_2_1.jsp
Allow: /webapp_2/page_2_2.jsp
[...]
Allow: /webapp_2/page_2_n4.jsp
Disallow: /

In all, I allowed 182 pages that way.

Using Google Search Console, I notice that 256 pages are marked as Blocked by robots.txt.
After checking them, I concluded that I indeed, do not want bots to crawl and index these pages because of the following reasons:

• CASE 1 - https://my_website.com/webapp/page.jsp?query_string (=> I allowed https://my_website.com/webapp/page.jsp (I would like bots to crawl and index it), why is the bot trying to crawl and index that page with a query string ? (Please note that I use these query strings in my webapps.))
• CASE 2 - https://my_website.com/webapp/page_not_allowed.jsp?query_string (=> I didn't allow https://my_website.com/webapp/page_not_allowed.jsp)
• CASE 3 - https://my_website.com/webapp/suppressed_page.jsp (=> I suppressed this page.)
• CASE 4 - https://my_website.com/page.jsp (=> page.jsp doesn't physically exist at that level, instead it exists here: https://my_website.com/webapp/page.jsp and is allowed.)
• CASE 5 - https://my_website.com/webapp/folder/doc.pdf (=> I don't need that PDF document to be crawled and indexed by bots.)
• CASE 6 - https://my_website.com/webapp/not_allowed_page.jsp (=> I didn't allow this page on purpose.)
• CASE 7 - https://my_website.com/ and https://www.my_website.com/ (=> I don't know if this is a problem since visiting these URLs leads to https://my_website.com/webapp_1/, setting I made on purpose.)
• CASE 8 - http://my_website.com/ and http://www.my_website.com/ (=> HTTP... my website is an HTTPS website.)

Shall I build robots.txt another way?
What can I do to avoid these issues?

Answer 1

The original specification is that robots "should," read should, read the robots.txt from the top to the bottom.

User-agent: *
Allow: /
[...]
Disallow: /

Should allow all pages on the site to be indexed. Robots.txt is not a site map. The disallow comes after the Allow and top to bottom the root plus all pages under the root are allowed.

/index.html is allowed
/directory/ is allowed
/index.html?key=value is allowed.

Case 1

Allowing https://my_website.com/webapp/page.jsp

Also allows https://my_website.com/webapp/page.jsp?query_string

This is correct. if you want to disallow query strings? disallow them before the allows in the robots.txt file.

User-agent: *
Disallow: /webapp/page.jsp?*
Allow: /webapp/page.jsp
Disallow: /

or disallow all query strings using a wildcard.

User-agent: *
Disallow: /*?*
Allow: /webapp/page.jsp
Disallow: /

Second, you should no that bots have nothing more than a moral requirement to honor the robots.txt files if they have read it and know a page should not be crawled.

It does not block bots from crawling pages disallowed in robots.txt file. If a link points to a page that the owner asked the bot not to crawl the bot may crawl the page to see what is there for the purpose of ranking the page, checking for bad things, or for whatever reason they desire.

But they really are not trying to find content you don't want indexed. There are trillions of pages on the internet. Google the biggest index only has 400 billion indexed.

If you need to disallow file types, like doc you can do that as well at the top of the robots.txt file

Disallow: /*.pdf

But changing the robots.txt files is not going to generate a request to have a file that is already in the search engine indexed removed. It will just no longer be updated.

To trigger a removal for a pdf file you would need to be through webmaster tools or by using a HTTP header, which can be done with .htacess. and do not block the file from being updated in robots.txt or the bot would never get the header telling it to not index the content.

<files *.pdf>
Header set X-Robots-Tag "noindex"
</files>

If content is on the site that is not to be public

Content on the site that should not be made public needs to be protected by password or other options provided by the server. Best to have such content in a sibling to the www directory so it does not have a URL to begin with.

But to block a directory with .htaccess

<If "%{REQUEST_URI} =~ m#^/site/includes($|/)#">
    Require all denied
</If>

which would allow applications like PHP or Jakarta to read and serve the content but prevent the content from being accessed directly.