Block ChatGPT and other AI Bots with robots.txt

from blog Robb Knight • Posts • RSS Feed, | ↗ original
Update 2024-03-30: Cory is maintaining a much better list on this repo which I will be using going forward. A robots.txt file "tells search engine crawlers which URLs the crawler can access on your site". Generally one would use this to tell a search engine which directories should be crawled, which ones shouldn't, or to completely block from...