All Questions
18 questions
3
votes
2
answers
5k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
- 102
8
votes
1
answer
791
views
Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)
Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says
the /fish rule matches any path that starts with /fish (...) it doesn'...
- 183
1
vote
1
answer
397
views
Should I add links to the Sitemap that are disallowed in robots.txt?
My robots.txt contains the line Disallow: /search.php.
Shall I nevertheless include http://example.com/search.php into my sitemap?
- 113
1
vote
1
answer
71
views
Robot.Txt File interpretation problem?
I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages:
{block name="frontend_robots_txt_user_agent"} User-agent: * {/block}
{block name="...
- 141
2
votes
1
answer
3k
views
Do I need to disallow mediapartners-google in robots.txt for SEO?
I found that most of the websites disallow mediapartners-google in robots.txt.
Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
- 155
1
vote
0
answers
9
views
Wordpress Website not indexing in google, no error on google webmaster [duplicate]
I just recognized my SEO Score is 0 and that my wordpress website is not indexed on google search engine, no automatic filling as well. I am not an expert, I never used a sitemap, nor htaccess but it ...
- 11
2
votes
1
answer
287
views
Should I prevent search engines from indexing empty user profile pages? If so, how much content is enough for indexing?
I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available:
The main user page (about me, last activity, ...)
Several book ...
- 281
3
votes
2
answers
162
views
Google claims not to find my robots.txt
Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it.
But there is a robots.txt file (https://soundhaus-band....
- 31
3
votes
1
answer
699
views
How to disallow Google crawling out of my website through images?
I'm taking care of a website which has multiple images coming from exterior links.
The thing is, those links are outgoing ones, and I want to avoid that as much as possible.
I tough of one solution ...
- 133
3
votes
1
answer
6k
views
Can I safely block Google AdsBot from spidering specific links?
My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different.
I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
- 281
2
votes
2
answers
73
views
How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?
After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
- 13.6k
2
votes
2
answers
146
views
Disallow Crawling of All Search Pages
Using robots.txt I am attempting to stop all crawling of search URLs
Disallow: /rest_of_url/search&tour*
Above is what I am using. Our URL looks like the following for all search results. ...
- 253
1
vote
1
answer
148
views
What are proper process to disallow a site from google crawl
I have done google and get the two solution for disallow whole a site from
google crawl.
1:
User-agent: *
Disallow:
2:
User-agent: *
Disallow: /
Now can anyone tell me which proper code ...
- 175
1
vote
3
answers
219
views
How do I remove a LOT of indexed pages from Google?
A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
- 11
6
votes
2
answers
6k
views
Serve a different robots.txt file for every site hosted in the same directory
We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
- 381
5
votes
4
answers
8k
views
Does Google ignore robots.txt [duplicate]
I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
- 153
0
votes
2
answers
167
views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress (...
- 113
3
votes
2
answers
5k
views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
morpheous