I'm trying to create a sitemap. I'm using the Screaming Frog software, but I think that this can be achieved in different ways with other software as well.

My links are generated with different parameters using php and they all look like example.com/content.php?id=123. I have a rewrite.php file that takes a clean url alias typed by the user (e.g. example.com/mi/nuevo/mensaje) and redirects the user to the proper ugly location.

For example:

example.com/mi-mensaje is read by the rewrite.php script and is displayed in the browser as example.com/content.php?id=523, or

example.com/mi-otro-mensaje is displayed as example.com/content.php?id=268, or

example.com/legislacion/mi-otro-mensaje is displayed as example.com/content.php?id=274

and so on...

PHP reads stuff from a MySQL database in order to know where to redirect the user.

Now, when I analyze the results with screaming frog, I am effectively told, some aliases can be indexed, getting a 200 status result, and some others cannot, getting a 301 or 302 status. The funny thing is that all aliases work the same way. Here's as example of what I mean, here are two aliases:

Here are two aliases

In the above screenshot I am comparing the selected (green) link, with the link immediately above it. As you can see the status is different even though these 2 links work normally.

Here is the resultant output for the above highlighted link:

Here is the ok one

And here is the output for other link that I am comparing it against.

the other one non indexable

As can be seen the 2 links are generated in the exact same way, but the second link is non indexable.

In both cases they do have a redirect URL. My question is why is the first link indexable and the second not, when both links are similarly generated. Why status 200 for one link and 302 for the other? I can access both links by their clean alias.

I use a rewrite rule in my .htaccess file in order to generate the links. Each user is sent to rewrite.php and from there -and according to the parameters in the path- the correct content is shown.

Here is my .htaccess file:

<IfModule mod_rewrite.c>
Options +FollowSymLinks
RewriteEngine On
RewriteCond %{SCRIPT_FILENAME} !-d
RewriteCond %{SCRIPT_FILENAME} !-f
RewriteRule ^.*$ ./rewrite.php
  • "In both cases they do have a redirect URL" - but they shouldn't. This URL should not be redirected - it should simply be rewritten (as indicated by the directives in your .htaccess file). That "Redirect URL" should be completely hidden from the crawler/visitor. "Why status 200 for one link and 302 for the other?" - Well, that's the million dollar question. If that is all you have in your .htaccess file then "something in your PHP code" would seem to be doing this?
    – MrWhite
    Aug 13, 2019 at 9:45
  • From your earlier question it seems that your PHP code is issuing 302 (temporary) redirects to the script that generates the page - if that is still the case then that would certainly be the "error" here.
    – MrWhite
    Aug 13, 2019 at 9:49
  • It is impossible to tell from what you posted where the redirect is coming from. As MrWhite points out, it is likely in your PHP code. You'd need to add more information including the snippets of PHP that could be doing redirects. Aug 13, 2019 at 10:49
  • Thanks for your comments. In my other question I've asked about SEO, and even if this one is related to the other one, they ask about diferent things (this one is about creating sitemaps). Abut my PHP code, the redirection is done using header('Location: '.$link);
    – Rosamunda
    Aug 13, 2019 at 11:29
  • Now, I've rephrased the question and I've just found the solution. How can I post it here so I can help others? It stills shows "on hold".
    – Rosamunda
    Aug 13, 2019 at 12:11


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.