Configurable robots.txt

[dariusk]

Pitch

This should be configurable by admins via the admin interface (so that admins using managed hosting can set these rules). For now it could be handled like the custom CSS interface: a <textarea> that lets you edit and save your robots.txt.

Probably a good idea to link to a robots.txt "cheat sheet" as well, containing common rules that you might want to implement.

Motivation

Communities should be able to set their own rules regarding web crawlers of all sorts.