4

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title"

I noticed the description that said the robots.txt is missing, but it was there and it looked like this:

User-agent: Google
Disallow:

User-agent: *
Disallow: /

This was made by my former co-worker, to avoid any misunderstandings. From the beginning I can say that to achieve what he wanted I think he should've placed the user-agent:google at the end.

Now I need help from you guys to help me create a proper robots.txt since I know it's not something you should just mess with if you don't know what you're doing.

I can come back with more screenshots of .. anything :)

I am the administrator of several webpages which are very very similar, but I've never encountered such a problem and never needed the robots.txt

3

Your user agent for Google wrong. It should be Googlebot. A user agent of Google is probably not recognized. See Google's "robots.txt" section of their help article Monitor crawling activity and errors.

2

Actually, if you want all crawlers to be able to crawl whole your website, you don't even need robots.txt, because if you don't have robots.txt file, crawlers would assume that they are allowed to crawl your website.

2

Sometimes Google crawls webpages before the proper title is setup so you might not have the right title displayed. That however goes away soon after the page is crawled again.

The message you are seeing basically says that this particular page is blocked by your robots.txt. Your former co-worked just blocked all web crawlers from getting to your site.

What you need to do is change your robots.txt to:

User-agent:*
Allow:/

Note: this will allow all crawlers to crawl your entire site. You can limit what is crawled by adding something like this:

User-agent:  *
Disallow: */site-section/

Then wait until the site is crawled again or go to your GWT and 'ask' google to crawls your site again. Below is a screen show:

Fetch as Google

If you want the whole site crawled then just hit the fetch button.

Note: changes might take a few weeks to take place.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.