1

This is my robots.txt file:

User-agent: * 
Disallow: /user/
Disallow: /user/*
Disallow: /home/*
Disallow: /adminpanel/*
Disallow: /Valorant_Guess/*
Disallow: /Leagueoflegends_Guess/*
Disallow: /Rocketlegue_Guess/*
Disallow: /Csgo_Guess/*
Disallow: /Apexlegends_Guess/*
Sitemap: https://www.example.com/sitemap.xml

My robots.txt file is visible at /robots.txt.

With a commenter's help: It looks like the problem is that my /robots.txt path is 302-redirecting to my home page when Lighthouse requests it (/), so Lighthouse never even sees my robots.txt. And that this issue is unrelated to the contents of my robots.txt file.

In addition to this, my webapp serves the robots.txt normally when visited by a web browser.

I coded my app on Flask using the gentelella package and the HTML/CSS/JavaScript is a Wella (paid template).

Here is the code of my robots.txt page in Flask: Edited (corrected the text/plain)

@blueprint.route("/robots.txt")
def robots():
  return send_from_directory(current_app.static_folder, 'robots/robots.txt', mimetype='text/plain')

And here is the Flask code for my home page:

@blueprint.route('/', methods=['GET', 'POST'])
def accueil():
    user_name = session.get("user")
    if (user_name):
        #... unrelated code ...
    return render_template('accueil/index.html' )

No redirects have been added in the Flask code.

Edit : It is not a server related error as Lighthouse has the same errors as when launch the webapp on a server or locally

here is a picture of the error : enter image description here

So how can I prevent my app from issuing an erroneous 302-redirect when Lighthouse requests my robots.txt file? It does not issue the same 302 redirect when visited normally in a web browser.

6
  • Your /robots.txt path is 302-redirecting to your home page when Lighthouse requests it (/), so Lighthouse never even sees your robots.txt (screenshot). This issue is unrelated to the contents of your robots.txt file. You'll need to look into your website's redirect configuration, and remove whatever redirect rule is causing the issue. Commented Mar 15, 2023 at 18:21
  • Specifically, you'll need to prevent your web server from 302-redirecting /robots.txt to / when Lighthouse requests it. For some reason, your website 302-redirects away from your robots.txt when Lighthouse requests it, but serves the robots.txt normally when visited by a web browser. Commented Mar 15, 2023 at 18:26
  • Also, your robots.txt file is being served with a text/html content type header, when it should be served with a text/plain content type header. This also points to an issue with the server configuration. Commented Mar 15, 2023 at 18:29
  • ok thank you. This is really helpfull for me !
    – szocel
    Commented Mar 15, 2023 at 18:45
  • Could nginx be the culprit? I set it up with ''' server { # listen on port 80 (http) listen 80; server_name _; location / { # redirect any requests to the same URL but on https return 301 https://$host$request_uri; } } server { # listen on port 443 (https) listen 443 ssl; .... '''
    – szocel
    Commented Mar 15, 2023 at 18:53

1 Answer 1

1

I found the culprit of redirects crawlers! In my flask app I used flask_paranoid. The package protects the application against certain attacks in which the user session cookie is stolen and then used by the attacker.

the command

paranoid.redirect_view = '/'

was redirecting all the crawlers.

For the moment I have no idea how to make them compatible with one another.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.