0

I have a php file that extracts a post id from a url's parameter to show a specific post content. for example it uses the postid parameter from the following url to show the post content with id of 21:
example.com/mydir/posts.php?postid=21
the postid is dynamic and every time that a user post a content a new post id will be added to the database.
how can I specify this php file in my robots.txt and sitemap.xml?
Currently, I use the following:
robots.txt: Disallow: /mydir/ Allow: /mydir/posts.php?username=* sitmap.xml:
<loc>example.com/mydir/posts.php?postid=*</loc>

2
  • Why are you disallowing /mydir/? Many robots don't understand Allow: rules. It would be more widely understood to just disallow the portions of /mydir/ that you don't want crawlers to hit. – Stephen Ostermiller Dec 18 '20 at 14:56
  • XML sitemaps don't support wildcards. Your XML sitemap should list many posts.php?postid= URLs. It should have a list of all the valid post ids. – Stephen Ostermiller Dec 18 '20 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.