For years, websites included information about what kind of crawlers were not allowed on their site with a robots.txt file.

Get access to this article and all of the in-depth coverage from the 247Sports Network with this special offer.