Questions tagged [disallow]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
35 views

Google indexing Wordpress in subdirectory disallowed in robots.txt

I have a headless wordpress site just using wordpress API. My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress. I set up robots.txt as follows: User-agent: *...
2
votes
1answer
39 views

Can I use robots.txt to allow Google to access some folders, but disallow everything else?

I want to allow Google bots to crawl only my index.php and the content of some folders. Would that work? User-agent: * Allow: /index*.php Allow: /folder1/ Allow: /folder2/ Allow: /folder3/ Allow: /...
1
vote
1answer
111 views

Google can't access my page due to robots.txt but the testing tool says it's OK

My robots.txt is the typical that allows all bots everywhere User-agent: * Disallow: / When I test it with Google tool everything is OK. But in Google Search Console I get an error that my site can'...
2
votes
1answer
398 views

How do I disallow some but not all subdirectories of a folder using robots.txt?

I have one folder named Frontend and under this folder I have subfolder names. Like: Images CSS JS Font I want to disallow below folder: /forntend/css/ /forntend/js/ /forntend/font/ but I don't ...
2
votes
1answer
83 views

Can I have both “disallow nothing” and “disallow something” statements in my robots.txt?

I had to make an update to an update to a robots.txt to disallow access to sub directory. There was already a disallow statement but i added another one, is the updated robots.txt file below correct ? ...