Fix links (#1387)

* Fix 404s

* Combine robots config and sitemap config

* Delete empty docs page

* Disallow partial pages

content/docs/details/custom-api-docs-renderers/index.md

@@ -53,7 +53,7 @@ Then click "Add Component" and set the type to ApiDocsWidget and the name to the
 
 ## Step 3: Publish your custom plugin
 
-Read [the docs on custom plugins](/docs/custom-plugins/configuring) then build your package and publish to artifactory.
+Read [the docs on custom plugins](/docs/custom-plugins/overview/) then build your package and publish to artifactory.
 
 In a nutshell:
 ```

content/docs/details/techdocs/troubleshooting.md

@@ -85,7 +85,7 @@ Larger TechDocs repositories can take some time to build when being updated or v
 
 There are two ways around this currently:
 
-1. Make sure your entity with the relevant TechDocs annotation is listed in the root `catalog-info.yaml` file of your repo and the docs are in a root `docs/` folder. If this is the case AND your admins have set up the [correct autodiscovery](docs/integrations/github-discovery/#steps-to-add-new-auto-discovery-paths) in Roadie for Github based repositories, changes to your docs should be built when they are merged to the default repository branch, so you won't have to wait for docs to build when viewing them.
+1. Make sure your entity with the relevant TechDocs annotation is listed in the root `catalog-info.yaml` file of your repo and the docs are in a root `docs/` folder. If this is the case AND your admins have set up the [correct autodiscovery](/docs/integrations/github-discovery/#steps-to-add-new-auto-discovery-paths) in Roadie for Github based repositories, changes to your docs should be built when they are merged to the default repository branch, so you won't have to wait for docs to build when viewing them.
 
 2. Your organisation can start [building and publishing TechDocs via your own CI pipelines](/docs/details/techdocs/build-via-ci) so that the latest version is always ready to view in Roadie. 
 

content/docs/tech-insights/track-sonarqube/index.md

@@ -113,4 +113,4 @@ To connect SonarQube with Roadie we will use a broker connection. The broker is
 
 ## Create checks with the data from the Sonarcloud Data Sources
 
-The process for creating checks and scorecards is very similar to the process for doing the same with SonarCloud. You can find details about how to do that [here](/tech-insights/track-sonarcloud/#create-checks-with-the-data-from-the-sonarcloud-data-sources). 
+The process for creating checks and scorecards is very similar to the process for doing the same with SonarCloud. You can find details about how to do that [here](/docs/tech-insights/track-sonarcloud/#create-checks-with-the-data-from-the-sonarcloud-data-sources). 

gatsby-config.js

@@ -57,6 +57,28 @@ const getContentfulOptions = () => {
   };
 };
 
+// These paths are blocked from search engine indexing and search engine access.
+const DISLALLOW_LIST = [
+  '/purchase/',
+  '/purchase/success/',
+  '/tailwind/404/',
+  '/installation-pending/',
+
+  // These are partials which eventually get embedded into other pages.
+  '/docs/details/create-proxy/structure/header/',
+  '/docs/getting-started/adding-a-catalog-item/structure/header/',
+  '/docs/details/create-proxy/basic/',
+
+  // These are partials which eventually get embedded into other pages.
+  '/docs/getting-started/adding-a-catalog-item/bitbucket/',
+  '/docs/getting-started/adding-a-catalog-item/github/',
+  '/docs/getting-started/adding-a-catalog-item/gitlab/',
+  '/docs/getting-started/adding-a-catalog-item/roadie-api/',
+  '/docs/getting-started/adding-a-catalog-item/aws-s3/',
+  '/docs/getting-started/adding-a-catalog-item/azure-devops/',
+  '/docs/getting-started/adding-a-catalog-item/roadie-cli/',
+];
+
 // Only environment variables prefixed with GATSBY_ are available in the runtime. Here we turn
 // a server side variable into a runtime one. This variable is later used to determine which
 // branch of a split testing experiment we are on.
@@ -187,7 +209,21 @@ module.exports = {
     {
       resolve: 'gatsby-plugin-sitemap',
       options: {
-        excludes: ['/purchase/', '/purchase/success/', '/tailwind/404/'],
+        excludes: DISLALLOW_LIST,
+      },
+    },
+
+    {
+      resolve: 'gatsby-plugin-robots-txt',
+      options: {
+        host: getSiteUrl(),
+        sitemap: `${getSiteUrl()}/sitemap-index.xml`,
+        policy: [
+          {
+            userAgent: '*',
+            disallow: DISLALLOW_LIST,
+          },
+        ],
       },
     },
 

package.json

@@ -38,6 +38,7 @@
     "gatsby-plugin-postcss": "^5.4.0",
     "gatsby-plugin-react-helmet": "^5.4.0",
     "gatsby-plugin-react-svg": "^3.0.1",
+    "gatsby-plugin-robots-txt": "^1.8.0",
     "gatsby-plugin-sharp": "^4.25.1",
     "gatsby-plugin-sitemap": "^6.13.1",
     "gatsby-plugin-twitter": "^4.4.0",