Issue #14601 new

Improve robots.txt on public repos for web crawlers

Jesse Yowell
staff created an issue

Sites like Github have a policy to include files/directories of the master branch on public repos for webcrawlers. We would need to change our robots.txt policy to include this on our site too.