GitHub's Mysterious robots.txt

from blog Ersei 'n Stuff, | ↗ original
↗ original
Recently, I came across GitHub's robots.txt. There are some interesting entries in there, so I'm going to try and find out why those entries are there. At the time of writing (2023-02-12 14:53 EST), GitHub's robots.txt looks something like this: # If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots # We...