Hot answers tagged

8

Err? India does have copyright laws... Sorry to say but your comment about India not having copyright laws is utter non-sense and unfounded. Furthermore its highly doubtful that your hard work is not being rewarded because Google penalises sites for copying identical, it can take some time but it happens. Scrapers use a range of online and local tools ...


5

MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt: User-agent: MJ12bot Disallow: /


4

From your comments on another answer, MJ12Bot is visiting your site less than once an hour (421 times in 25 days.) The best thing to do is to not worry about it. Crawl-Delay is useless for you because no crawler will obey a craw-delay so large.


4

You can use something like this code in your DOCUMENT_ROOT/.htaccess file: RewriteEngine On RewriteCond %{HTTP_HOST} ^(?:www\.)?subdomain\.com$ [NC] RewriteCond ${ipmap:%{REMOTE_ADDR}} !^(127\.0\.0\.1|192\.168\.|10\.|1\.2\.3\.4)$ RewriteRule ^ - [F] Source: answer from anubhava on Stack Overflow You can also allow specific IP addresses and deny the rest ...


4

I was pretty much in the same boat, save for the fact that mine was not an issue with spam or viruses. Instead, I had loads of broken links from when the domain was owned by someone else. Patience My advise to you is to be patient. As you say, you have already notified Google about the problem and they have cleared and approved the domain. Your site ...


4

This is the very interesting situation as you didn't know the history of the domain. There are several things you may need to look. Virus vs Spam: Check if it had infection issue or it was penalised by Google due to poor SEO practice. If only virus issue then it's not that hard to fix, send the request and generally Google updates in about 24 hours if the ...


3

This is due to the number of people who embed these HTTP libraries in their own software for the purpose of scraping content from other sites, which is often done for the purposes of copyright infringement. Well-made crawlers which are legitimate and designed for a specific purpose (like archiver bots and search bots) have their own custom user agent strings ...


3

I want to avoid large scale content theft and plagiarism‎ which mainly comes from India as they have no laws preventing Indians from stealing content. I don't understand how you come across the conclusion that india is the only source that is accessing your site to steal content. I'm gonna apologize here for a minute because we must make an assumption that ...


3

/404.html?page=/wp-content/uploads/2020/03/... I assume the ... is "anything that follows". To block this URL, you could use mod_rewrite in your root .htaccess file. But you need to use a condition (RewriteCond directive) to match the query string. The RewriteRule pattern matches against the URL-path only (less the directory prefix), ie. 404.html. To match ...


2

Is there a way to quiet down MJ12Bot ambitions The MJ12Bot reportedly obeys robots.txt and the (non-standard) Crawl-Delay directive: How can I slow down MJ12bot? You can easily slow down bot by adding the following to your robots.txt file: User-Agent: MJ12bot Crawl-Delay: 5 Crawl-Delay should be an integer number and it signifies number of ...


2

Here is my take on this: Firstly, ask the "legal team" to explain to you what a cookie is, and how that law protects users. Remind the "legal team" that the out-of-touch "politicians" who made that "law" didn't actually understand what a cookie is either, what a cookie shares around, and therefore didn't understand how futile their "law" would be at ...


2

They are just scanning the network for open proxy. Your server is safe. But better report them to their ISP before there is a victim.


2

This sounds like a server firewall issue. Failed ssh attempts can usually trigger a rule that will then block your IP. 1und1.de has a guide for how to manage their server firewall (I had to use Google Translate to read it). You may be able to log in to your server control panel to whitelist your IP. If not, you could try getting in touch with 1und1.de ...


2

You'll never be able to block malicious users of an entire country. Likely, they are using automated methods to scrape your content, so I'd suggest making your site really annoying and time consuming to scrape if youre really worried. Otherwise, such is the nature of the internet.


2

This message looks to me like it comes from Symantic's Web Gateway. Here is a page where somebody with the same error message points the finger at Symantec. Symantec has a URL filtering categories page that lists the category your website has been misassigned: Weapons (plus Military) -- This category contains Web sites from military organizations/...


2

My bad: My httpd configuration redirects 403 errors to a custom error page, but apparently doesn't report it as an error. Since robots.txt doesn't exist, httpd dutifully did what it was supposed to and returned my custom error page instead. Hopefully this helps someone else.


2

Did you received the message via email or via Search Console? if it is via email ignore. If message is on search console, try to overwrite it by adding a robots.txt to allow all bots to crawl your website, like for example: User-agent: * Disallow: Also check that no meta robots tags are present (check every page) if the following appears and remove it: &...


2

The types of bots One way of thinking about bots is to treat them as following the Lawful-Chaotic and Good-Evil spectrum. (Although bots cannot be evil, they can be instructed to act on behalf of people doing evil.) An example for Chaotic good is Shodan, which helps improve security by helping research and checking your own systems, even if other people ...


2

If the new website has been developed independently from the infected website then you may be able to delete the files on the server and the database and then upload the new website cleanly without infection. If the new website has occupied the same hosting account then it is likely also infected. You can use a commercial service such as https://sucuri.net ...


1

This is a question that I've encountered from clients multiple times, most of the time it's by people with little knowledge of PPC, who are in highly competitive fields but on very tight budgets, and who tend to be more than a little paranoid. It's also usually from people who would personally do exactly the same thing to their competition. However, that's ...


1

Take a look at this article on blocking AWS with .htaccess. The solutions there is to add the following to your .htaccess file: RewriteEngine On RewriteCond %{HTTP_REFERER} ^http://.*amazonaws\.com [OR] RewriteCond %{REMOTE_HOST} ^.*\.compute-1\.amazonaws\.com$ [NC,OR] RewriteCond %{HTTP_USER_AGENT} "AISearchBot" [NC,OR] RewriteCond %{...


1

Unfortunately at this time Microsoft does not have a global "Report a False Positive" feature which would allow your website to be scanned, re-assessed and marked as safe for all 365 users. There is however an option that your user can do and that is request from their IT department to add your site to their custom whitelist (https://support.office.com/en-us/...


1

You say, new website is placed it on a temporary separate domain. There is a clear cause on the test page: "domain is blacklisted by Google Safe Browsing". Beside of this, what means "new website we developed on Joomla 3.9". What exactly do you developed? Template? Extensions? Hacked in the core? I would say, make on the clean domain a clean Joomla ...


1

Afaik, this must be done on a campaign by campaign basis. While still tedious because it needs to be done for each campaign, if not already using the Google Ads editor, you could use it to make the campaign edits and then upload/push the campaign changes all in one go, rather than trawling through the Google Ads interface campaign, by campaign. Google Ads ...


1

Allowing certain bots will cause some troubles for you. For instance, you will be blocking Alexa, Archive, etc. robots from accessing your website. I would recommend blocking bad robots by user-agent. Important note Bad bots can change their user-agents in request header. So this won't completely safeguard your website against bad crawlers.


1

Blocking a country is complicated and the other answers posted here explain the drawbacks such as thieves scraping your site from a proxy server outside their country, etc. However to answer your technical question here are two ideas to explore. The Easy Way I don't know what kind of site(s) you operate but static sites can go on AWS CloudFront and you may ...


1

If you are using Apache, edit the .htaccess file in your website root directory and add the below. 192.0.2.123 will represent the IP address of the user you wish to block. order allow,deny deny from 192.0.2.123 allow from all If you are using NginX, edit the vhosts config file (location depends on your setup) and add the below. location / { deny 192....


1

You have to wait for their database to be updated with your site. They provide instructions on how to request they add a site.


1

The first thing that I would check is DNS. It sounds like you recently changed your DNS settings and it has not completely propagated yet. When you change DNS settings it may take several days for the changes to fully take effect depending on your time to live (TTL) settings. Some computers will see the new info sooner than others. Computers that have ...


1

I added google analytics to the mobile version of my site and three days later it seems that most of the visitors either do not have a language set or a country code set. First be aware that GA only gives you a "processed" view of your actual traffic. It's purpose is to present you with statistically relevant data and only statistically relevant data. It is ...


Only top voted, non community-wiki answers of a minimum length are eligible