Improve robots.txt on public repos for web crawlers

Issue #14601 open
Jesse Yowell
staff created an issue

Sites like Github have a policy to include files/directories of the master branch on public repos for webcrawlers. We would need to change our robots.txt policy to include this on our site too.