[SEO Audits] Document has a meta description

[rviscomi]

Audit group: Content best practices
Description: Document has a meta description
Failure description: Document does not have a meta description
Help text: Meta descriptions may be included in search results to concisely summarize page content. Read more in the Search Console Help page.

Success condition: Query selector head > meta[name=description] exists and has a non-empty content attribute.

Notes

• If short-circuiting the audit when nosnippet exists, use the audit description field to explain that the meta description is unused. Other robots directives that affect the meta description like noindex are covered in separate audits. (not used)

[kdzwinel]

Hi all 👋 I started working on this issue. Can I get assigned please? 😉

FYI there are couple of corner cases regarding "nosnippet" that are not mentioned in the task description:

• 'robots' metatag can have multiple values (e.g. noarchive, nosnippet)
• there can be multiple <meta name="robots" on a page
• name="robots" can be replaced with a bot specific name="googlebot"
• directives can also be sent via X-Robots-Tag HTTP header
  • there can be multiple of these headers
  • they can have multiple values X-Robots-Tag: noindex, nofollow
  • they can refer to a specific bot X-Robots-Tag: googlebot: nofollow

Other notes: everything is non-case sensitive, commas are required, spaces after commas are optional

Sources:

• https://developers.google.com/search/reference/robots_meta_tag
• https://webmasters.googleblog.com/2007/03/using-robots-meta-tag.html
• http://searchengineland.com/meta-robots-tag-101-blocking-spiders-cached-pages-more-10665

[kdzwinel]

Per our discussion today, name="googlebot" and googlebot: are out of the picture for now since we are not focusing on search engine specific directives. This brings up a question: should we really support nosnippet since it's recognized only by a googlebot (according to this)?

[rviscomi]

Yeah let's remove the nosnippet condition. I'll update the audit description.

I checked the HTTP Archive data and that particular directive is only present in ~150 pages (of the 500k measured).

[kdzwinel]

👌

BTW I'm not sure how much we can depend on httparchive data regarding SEO, their robot also follows the "robots" directives so the results may be skewed. On the other hand, if their robot does follow these directives, how did you get "noindex, nofollow" on the results of that query? 🤔

[rviscomi]

HTTP Archive runs as a totally separate crawl from the Internet Archive's Wayback Machine. We've talked about respecting robots directives but they're ignored for now.

[kdzwinel]

For the record: we've decided to trim the description before evaluating its length and to provide debugString's for audit failures (tag not found, empty value).

#3227 (comment)