The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek at it!
Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if ...
Apache Tomcat has gzip filter and it starts to zip from 2kb, my quick test tells that it's the lowest boundary, and you can increase it at least to 3-4kb. Because for 2kb you will get a similar size at the output + time penalty for zipping.
Content Security Policy (CSP)
Quick Reference Guide
CSP-Reporting on Google Developers
If you're just starting out with CSP, it makes sense to evaluate the
current state of your application before rolling out a draconian
policy to your users. As a stepping stone to a complete deployment,
you can ask the browser to monitor a policy, reporting violations but