New answers tagged web-crawlers
0
The only thing that you need for Google to follow every link is to let Google know that your site exists. This can happen "naturally" if your site is linked from sites that already exist in Google index, alternatively you can use their site submission form to explicitly add your site to the list of sites to index here.
robots.txt and sitemap.xml are not ...
1
Of course not. These two files are not required all the time.
In Google Webmaster Tools, Google says that a robots.txt is not necessary if you want that all webpages being crawled. If you want to put only these following lines in a robots.txt:
User-agent: *
Allow: /
Just don't create a robots.txt.
Moreover, a sitemap.xml helps indexing of your website ...
3
No, neither is required.
By default your site will be indexed by Google. The robots.txt file is useful for preventing it from accessing specific directories or files. It's not a security mechanism, however, and if you don't want the public to access those pages/files, you should block access another way, such as an .htaccess file.
The sitemap.xml is also ...
0
I think in order for your page to get index by search engines like Google, Bing etc. you need to have both robots.txt and sitemap.xml files.
A quick Google shows that your robots.txt file has to be placed on root directory for your website. This link should give you more details about how-to create that file and what to do with it.
And for sitemap.xml, ...
0
FYI I also received a DOS attack from google-proxy:
host 66.249.82.43
43.82.249.66.in-addr.arpa domain name pointer google-proxy-66-249-82-43.google.com.
log extracts:
Jun 9 21:19:43 gemelos kernel: PAX: From 66.249.82.43: execution attempt in: (null), 00000000-00000000 00000000
Jun 9 21:19:43 gemelos kernel: PAX: terminating task: ...
0
Googlebot would NOT be able to crawl such a website. Googlebot does not use cookies, so it would not maintain a session to be able to crawl.
If the sessionid were in the url, Googlebot would still have trouble because it would get different sessionids each time it crawled the home page and you would have massive duplicate content problems.
Furthermore, ...
1
The problem you have isn't Bing but the way your server is handling error responses.
Your 404 pages are reporting: SERVER RESPONSE: HTTP/1.1 200 OK
It should be reporting: SERVER RESPONSE: HTTP/1.0 404 NOT FOUND
So search engines are assuming they are valid pages and that's why they are being crawled all the time. Fix this and Bing should start to stop ...
Top 50 recent answers are included
