For years, websites included information about what kind of crawlers were not allowed on their site with a robots.txt file.

Companies like Canva, GoDaddy, and Instacart are also exploring how they can use this technology in their platforms.