feat(v2): Add themeConfig.noIndex option #3528

[hamzahamidi]

Motivation

This PR is intended to add the noIndex option which was present in V1: https://docusaurus.io/docs/en/site-config#noindex-boolean
More details in the correspondent issue #3528

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

The feature is to add a noIndex option in docusaurus.config.js. If true <meta name="robots" content="noindex" />
will be added to the template.

Related PRs

[docusaurus-bot]

Deploy preview for docusaurus-2 ready!

Built without sensitive environment variables with commit 691b703

https://deploy-preview-3573--docusaurus-2.netlify.app

[slorber]

Thanks for the PR, looks like a good start.

Instead of adding the noindex meta from themes code, what about injecting the tag in a generic way directly in core (like, through the SSR template). This way we are sure that this feature work consistently across all themes, including themes built by users/community

Also, we have a migration cli. If v1 had the noIndex option, that would be nice if running the migration cli did output a v2 website also having the noIndex option. You can test this migration with yarn test:v1Migration:migrate

Added some code comments too.

[slorber]

LGTM, just need to remove the noIndex setup for development mode and good to merge, thanks

[slorber]

Thanks @hamzahamidi , that looks nice ;)