Tag Info

New answers tagged

1

Alas if your data is worth getting it will be gotten. When we decided to handle IP addresses, the scrapers went horizontal on us and came back with thousands of IP addresses, each accessing us well under the limit we set. I am myself hunting for the newest hints on how to stop robots that do not respect any protection.


0

Apart from providing your results via the ajax method, you could also interpret the search query from the URL parameters or URL structure (via apache aliasmatch ?). e.g example.com/zoo.php?q=unicorns or example.com/zoo/jellytiger and then return a page with the data already in the html, as if a user would have just typed unicorns or jellytiger in the ...


0

You can use display: flex and then specify an alternative order: n value for divisions that moves your menu division above or below other divisions. In other words, you don't need two menus with the same content if you always hide one or other of them. A guide to FlexBox W3C Flexible Box Layout


1

Yes. You can. I do not know the tool you are using, but here are some clues. Not a search engine (duh!). High number of requests over a longer period of time- more than a user. Does not request an image, CSS, or JS (some do, most don't). Changes agent name and OS over a longer period of time. Comes from a web-hosting IP address block/range. Check for ...


0

Site search and search results shoud never be crawlable, regardless of whether they are AJAX or not. See: Matt Cutts: search results in search results. Google penalizes sites that try to get their search results indexed. Googlebot is now starting to execute JavaScript and index any text that gets written into the page. I have a random password generator ...


0

Just a quick note that a 500 internal server error is an issue with your actual server. If you want to test this in the future, then you can run this command in terminal to see what your server is returning: curl -v -H 'Host: yourdomain.com' yourserver IP (replace with your actual domain & your server IP address in those fields).


1

Create a robots.txt in a bucket. Create another origin for your cloudfront distribution. Set your bucket's priority higher then your website. Invalidate your site's robots.txt on Cloudfront. After doing the above, Google will read the sites robots.txt when crawling your site and will get to see the different robots.txt when following links from your cdn. ...



Top 50 recent answers are included