Tagged Questions
2
votes
2answers
156 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
2
votes
3answers
164 views
Hiding php includes from search spiders?
Quick and simple question.
I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are ...
2
votes
9answers
4k views
Need to Stop Bots from Killing my Webserver
I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
1
vote
2answers
254 views
Robots.txt not working because my site is still searchable
I put a robots.txt in my server
This is the code that I putted in
User-agent: *
Disallow: /
I just want my entire site won't be searchable in google.
I also put in the NO INDEX, NO FOLLOW on the ...
3
votes
5answers
2k views
Disallow robots.txt from being accessed in a browser but still accessible by spiders?
We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
