All Questions
4 questions
3
votes
1
answer
149
views
What is a robots.txt file and how can I use it
I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create ...
1
vote
1
answer
908
views
Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files
Here is a picture of my Google Search Console:
When I try to click at the url for example error 18: wp-content/uploads/2016/01/
Appear File not found.
Of course inside the path (wp-content/uploads/...
- 31
1
vote
1
answer
273
views
Unexplainable non-existing page requested for months
While I am quite familiar with bots requesting well-know pages as used for WordPress and phpMyAdmin on any site the can reach, I'm now dealing with a bot requested a page I really can't explain.
For ...
- 111
1
vote
2
answers
583
views
Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?
My robots.txt:
User-agent: *
Disallow: /
A page two directories below the root is being hit by a Yahoo bot and getting a 404:
HTTP_REFERER: [empty string]
HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
- 221