In the robots.txt file, under

User-agent: *
User-agent: Googlebot
# disallow the following specific files

there are the following lines. I don't understand the two Disallow lines:

Allow: /*.html
Disallow: /*URL
Disallow: /*.htmlURL

In some 404 log entries, I've seen a 404 error for filename.html immediately followed by a URL address that isn't on my site. Don't know what this gets anybody by appending a different URL to the end of the initial one. But I'm wondering if the above Disallow lines in robots.txt are targeting that.

  • Disallow: /*URL - Is URL the literal text "URL" or some specific URL as you appear to refer to later in your question? Is this your site?
    – MrWhite
    Nov 22, 2022 at 2:09

1 Answer 1


Looks like somebody was copying robots.txt examples without thinking about what they were doing.

As per random hits to disallowed pages, hackers often use robots.txt to find sensitive files, entries to the admin panels for further exploitation and so on.

Not necessarily manually. There are bots that do all this early investigation and simple exploitation automatically.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.