Robot.Txt File interpretation problem?

Question

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages:

{block name="frontend_robots_txt_user_agent"} User-agent: * {/block}

{block name="frontend_robots_txt_disallows"} Disallow: {url controller=compare fullPath=false}

Disallow: {url controller=checkout fullPath=false}

Disallow: {url controller=register fullPath=false}

Disallow: {url controller=account fullPath=false}

Disallow: {url controller=address fullPath=false}

Disallow: {url controller=note fullPath=false}

Disallow: {url controller=widgets fullPath=false}

Disallow: {url controller=listing fullPath=false}

Disallow: {url controller=ticket fullPath=false} {/block}

{block name="frontend_robots_txt_allows"} Allow: {url module=widgets controller=emotion fullPath=false} {/block}

{block name="frontend_robots_txt_sitemap"} Sitemap: {url controller=index}sitemap_index.xml {/block}

{*
    @deprecated

    Will be removed in 5.6 without alternative
*} {block name="frontend_robots_txt_sitemap_mobile"}{/block}

Is there any problem here with this files? I'm so strange with these lines.

Answer 1

The template file you have posted from your backend end may render like this

User-agent: *
Disallow: /compare
Disallow: /checkout
Disallow: /register
Disallow: /account
Disallow: /account
Disallow: /note
Disallow: /widgets
Disallow: /listing
Disallow: /ticket
Allow: /emotion

The above interpretation of your robots.txt indicates that most of the pages in your website are disallowed for crawling which means google will not crawl those pages

The first line User-agent: * means that any crawler including google should obey the following lines The Disallow: /checkout line means that the crawler should not crawl or index this url and similarly Allow: /emotion would mean that the crawler should crawl and index the page

Depending on your requirements you should accordingly set the urls you don't want google to crawl in the Disallow directive and the ones google should crawl in the Allow directive.

Just go through my interpretation of your robots.txt mentioned above and change your urls according to ones that exist in your website and according to your crawling requirements