About NortheasternSysNetBot/2.0

Hello! We are a team of computer science researchers at Northeastern University crawling popular websites on the internet to assess the security and performance of the HTTPS protocol. Our crawls are intended to be innocuous in nature, and we do plan to promptly contact the website administrator(s) if we find critical issues in any website during our study.

We follow good Internet citizenship guidelines by:

1. Respecting websites' Robots.txt specifications.
2. Crawling a maximum number of 250 unique webpages from a single website (though due to the nature of our experiments, the crawler might access each webpage a couple of times).
3. Not crawling more frequently than is necessary for our research purposes.
4. Spreading the crawl load over time and over multiple destinations by randomly changing the subset of websites our crawler visits after every few seconds.

Please contact us at paracha.m@husky.neu.edu for any feedback, or if you would like to opt-out from our crawls.

Best,
Talha.