Linked Questions

2
votes
1answer
75 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
1
vote
1answer
48 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
0
votes
0answers
15 views

When will search engines disregard rules in my robots.txt to NOT crawl my website and files?

I'm working with a marketer that wants to set up a website with a text field for people type in their guess of a url for another publicly accessible but secret website. Let's say that publicly ...

15 30 50 per page
1
2