Questions tagged [disallow]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
37 views

Google is indexing URLs with parameters that are disallowed in robots.txt despite canonical URLs without parameters listed in the sitemap

All of my webpages are showing ?mode=grid & ?mode=list in the google coverage. But submitted sitemap shows normal URLs. For example: example.com/page/?mode=grid example.com/page/?mode=list ...
2
votes
1answer
88 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
0
votes
1answer
148 views

Google indexing Wordpress in subdirectory disallowed in robots.txt

I have a headless wordpress site just using wordpress API. My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress. I set up robots.txt as follows: User-agent: *...
2
votes
1answer
43 views

Can I use robots.txt to allow Google to access some folders, but disallow everything else?

I want to allow Google bots to crawl only my index.php and the content of some folders. Would that work? User-agent: * Allow: /index*.php Allow: /folder1/ Allow: /folder2/ Allow: /folder3/ Allow: /...
1
vote
1answer
147 views

Google can't access my page due to robots.txt but the testing tool says it's OK

My robots.txt is the typical that allows all bots everywhere User-agent: * Disallow: / When I test it with Google tool everything is OK. But in Google Search Console I get an error that my site can'...
2
votes
1answer
648 views

How do I disallow some but not all subdirectories of a folder using robots.txt?

I have one folder named Frontend and under this folder I have subfolder names. Like: Images CSS JS Font I want to disallow below folder: /forntend/css/ /forntend/js/ /forntend/font/ but I don't ...
2
votes
1answer
97 views

Can I have both “disallow nothing” and “disallow something” statements in my robots.txt?

I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ? ...