New answers tagged web-crawlers
Alas if your data is worth getting it will be gotten. When we decided to handle IP addresses, the scrapers went horizontal on us and came back with thousands of IP addresses, each accessing us well under the limit we set. I am myself hunting for the newest hints on how to stop robots that do not respect any protection.
Apart from providing your results via the ajax method, you could also interpret the search query from the URL parameters or URL structure (via apache aliasmatch ?). e.g example.com/zoo.php?q=unicorns or example.com/zoo/jellytiger and then return a page with the data already in the html, as if a user would have just typed unicorns or jellytiger in the ...
You can use display: flex and then specify an alternative order: n value for divisions that moves your menu division above or below other divisions. In other words, you don't need two menus with the same content if you always hide one or other of them. A guide to FlexBox W3C Flexible Box Layout
Yes. You can. I do not know the tool you are using, but here are some clues. Not a search engine (duh!). High number of requests over a longer period of time- more than a user. Does not request an image, CSS, or JS (some do, most don't). Changes agent name and OS over a longer period of time. Comes from a web-hosting IP address block/range. Check for ...
Just a quick note that a 500 internal server error is an issue with your actual server. If you want to test this in the future, then you can run this command in terminal to see what your server is returning: curl -v -H 'Host: yourdomain.com' yourserver IP (replace with your actual domain & your server IP address in those fields).
Top 50 recent answers are included