1

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages:

{block name="frontend_robots_txt_user_agent"} User-agent: * {/block}

{block name="frontend_robots_txt_disallows"} Disallow: {url controller=compare fullPath=false}

Disallow: {url controller=checkout fullPath=false}

Disallow: {url controller=register fullPath=false}

Disallow: {url controller=account fullPath=false}

Disallow: {url controller=address fullPath=false}

Disallow: {url controller=note fullPath=false}

Disallow: {url controller=widgets fullPath=false}

Disallow: {url controller=listing fullPath=false}

Disallow: {url controller=ticket fullPath=false} {/block}

{block name="frontend_robots_txt_allows"} Allow: {url module=widgets controller=emotion fullPath=false} {/block}

{block name="frontend_robots_txt_sitemap"} Sitemap: {url controller=index}sitemap_index.xml {/block}

{*
    @deprecated

    Will be removed in 5.6 without alternative
*} {block name="frontend_robots_txt_sitemap_mobile"}{/block}

Is there any problem here with this files? I'm so strange with these lines.

  • Is this the content of the file accessible at https://your-domain.example.com/robots.txt, or is it the content of the template file in the backend? – unor Apr 25 at 12:18
  • @unor this is in back-end folder called index.tpl – Mouna Mokhiab Apr 25 at 22:19
  • 1
    Okay. For a question about your robots.txt, you should include the generated robots.txt file, not the template file. The template file is relevant if you want to change something, but search engines will of course never see this template file, only the generated file. – unor Apr 25 at 23:11
  • @unor what is your proposed content of robot.txt? and I should replace robot.txt file in the same directory or not? (Please submit as an answer) – Mouna Mokhiab Apr 26 at 9:51
2

The template file you have posted from your backend end may render like this

User-agent: *
Disallow: /compare
Disallow: /checkout
Disallow: /register
Disallow: /account
Disallow: /account
Disallow: /note
Disallow: /widgets
Disallow: /listing
Disallow: /ticket
Allow: /emotion

The above interpretation of your robots.txt indicates that most of the pages in your website are disallowed for crawling which means google will not crawl those pages

The first line User-agent: * means that any crawler including google should obey the following lines The Disallow: /checkout line means that the crawler should not crawl or index this url and similarly Allow: /emotion would mean that the crawler should crawl and index the page

Depending on your requirements you should accordingly set the urls you don't want google to crawl in the Disallow directive and the ones google should crawl in the Allow directive.

Just go through my interpretation of your robots.txt mentioned above and change your urls according to ones that exist in your website and according to your crawling requirements

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.