I'm troubleshooting a website that uses an email verification process that has had a number of its tokens indexed by BingBot.
The process involves sending the user a confirmation link which contains a guid associated with an email.
The link directs to an account verification action similar to:
~/Account/[email protected]&token=0000-1111-2222-9999
While trawling server logs to investigate an unrelated issue I saw BingBot was crawling these URLs.
The site had no robots.txt to prevent indexing of the URLs for verification and resending confirmations. I've also noticed the site passes quite a few parameters in query strings.
Even so, I can't understand how these links have ended up getting indexed.
I spoke to a colleague who suggested the links may have been security scanned and indexed as they have passed through Microsoft mail hubs.
I'm unsure about this, and it doesn't explain the emails that come from non Microsoft domains.
The only other thought I had was the page gets indexed as the user hits it at confirmation, but colleagues have suggested this is unlikely - there must be a link to these verification URLs on the web somewhere for them to get crawled?
Some things to note:
Once email addresses have been confirmed - there's no real harm in these URLs getting hit again, the tokens are dead nothing gets reset or updated.
I am able to put some sensible changes in place to mitigate against this, but I'm just trying to understand how these have come into Bing's domain.
