This repository has been archived by the owner on Jan 17, 2023. It is now read-only.
Remove robots.txt and replace with <meta name=robots value=noindex> #2806
Assignees
Milestone
Comments
This was referenced May 4, 2017
Closed
ghost
mentioned this issue
ianb
added a commit
that referenced
this issue
May 15, 2018
This was intended to be fixed in #2806, but was only put in place for expired pages instead of all pages as intended.
jaredhirsch
pushed a commit
that referenced
this issue
May 15, 2018
ianb
added a commit
that referenced
this issue
May 15, 2018
This was intended to be fixed in #2806, but was only put in place for expired pages instead of all pages as intended.
testeaxeax
pushed a commit
to testeaxeax/screenshots
that referenced
this issue
Jun 7, 2018
…ges (mozilla-services#4460) This was intended to be fixed in mozilla-services#2806, but was only put in place for expired pages instead of all pages as intended.
chenba
pushed a commit
to chenba/screenshots
that referenced
this issue
Jun 26, 2018
This was intended to be fixed in mozilla-services#2806, but was only put in place for expired pages instead of all pages as intended.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
/robots.txtblocks all access by all robots, including friendly robots like the Twitter summarization bot (that turns a link into a nice card). I don't believe we can fix robots.txt unless we choose a small number of services to allow, and that doesn't align well with Mozilla's values. Plus it's annoying to maintain.Our goal is to avoid shot pages being indexed by search engines. We can accomplish this with:
We should add that to shot pages.
/shotsis not public, and I don't believe any other pages are meaningfully private.If we want to suppress indexing of images we'd have to add
X-Robots-Tag: noindexto those requests. At this moment (maybe just due to oversight) we don't block indexing of images.The text was updated successfully, but these errors were encountered: