[DISCUSS] Where to put development docs?

[cjcenizal]

Documentation on Kibana's architecture and how to contribute code to Kibana is currently split between elastic.co and READMEs in the repo. This creates confusion for contributors and consumers because it's not clear where new content should be added or where to look to find answers.

Elastic.co examples

• How to develop functional tests
• How to develop visualizations
• Managing dependencies
• Modules and autoloading

Repo examples

• Vislib overview
• ToastNotifications overview
• Contributing guidelines
• Styleguides

Pros of elastic.co

• Content is easily searchable
• Consumers looking for solutions in our docs will realize that building their own plugin or contributing a PR is also an option
• Feel free to add more

Pros of the repo

• Documentation is near the code being documented
• Developers are more likely to discover it in the repo than on elastic.co
• Content is formatted more nicely
• Content can be written in markdown which most developers are more familiar with
• No need to build docs to confirm edits appear as expected (both for developer and reviewer). Keeping changes in the repo reduces context switching overhead. (added by Stacey)
• Feel free to add more

How to vote

Elastic.co	Repo
👍	👎

[timroes]

To be honest I would most likely have it placed in some good developer documentation system, that supports proper Rest API documentation and JavaDoc places in one integrated system. Since we don't have that, and I haven't seen good alternatives yet, I rather read docs in Markdown in our repository, than on Elastic.co.

[stacey-gammon]

I added https://github.com/elastic/kibana-team/issues/12#issuecomment-408111801 to keep track of this pain point. I'm with Tim - I would much rather have a third option, but until there is time for that, Repo is much easier to maintain as a developer. Building/editing docs takes me out of my normal workflow and adds context switching overhead.

[cjcenizal]

I think that there would be a compelling argument for keeping all of our docs on elastic.co if we officially supported third-party plugin development. But considering that we don't, I think it sends a confusing message to our users to officially publish docs to do things we officially don't support.

Keeping all of our docs in markdown on GitHub will also make it easier to migrate them to a tool such as Tim's, which supports markdown. EDIT: Not such a good point, since it could support asciidoc too.

[snide]

+1 for repo if it's about development.

I do want to bring up a major frustration I have working with Kibana from a docs/dev perspective. The design team normally has to touch nearly every piece of Kibana which means we need to run it with everything on. I'd really love if we could maintain docs that described not just how to get local running, but how to get local running with all of the various systems running. Beyond that, it would be awesome to have some sort of one-liner that load everything up together

Some examples

1. Getting beats monitoring running
2. Getting logstash monitoring running
3. Getting logstash pipelines viewable
4. Getting the logstash pipeline viewer visable
5. Getting the various license states running

Right now our docs are written to setup the very bare-bones version of kibana. I'm sure we all have tons of aliases and scripts we all manage locally, but right now local Kibana dev feels like a lot of word-of-mouth. I feel bad constantly pinging people to set this kind of stuff up.

It'd be nice if we rewrote our dev docs to consolidate this type of pre-development setup in one place.

[stacey-gammon]

@snide - I found the fastest way to get most of those things up and running is through the Kibana UI itself, in the Add Data modules. It's super useful and if anything is wrong with the instructions, it's a bug and needs to be addressed.

That doesn't give us a single command, but it at least is a central place for instructions. Eventually we want to make that all doable directly in Kibana itself, so no going to the command line at all.

[epixa]

I find this issue a little confusing, to be honest. I think I disagree with most of the concerns folks have raised in the comments and at times with the whole premise of the issue itself. Our docs are in the repo today, and they are also published on elastic.co/docs. To be clear, this doesn't seem to be a proposal about documentation location particularly but rather about removing them from our website, scattering them throughout the code base, and changing the format from asciidoc to markdown.

By far the biggest benefit to putting the docs on elastic.co is discoverability. It's hard to find any of the developer documentation that exists in our repo unless we explicitly call it out in our contributing guide, which is now so large it is unbearable to use, yet the sum total of information we actually have documented via the contributing guide is minuscule relative to the amount of documentation we should have for developers.

I'm just going to refute a bunch of stuff in a massive wall of text for now, but I'll respond more productively when I get back from vacation:

Documentation is near the code being documented

There's merit to this idea, but it's only valuable if you already know where to look, and even if you know where to look, you aren't necessarily getting a complete picture in any single place in our code. For example, documenting vislib is a great start, but it's only valuable if you understand how visualizations work in Kibana, which is logic that is scattered across who-knows-how-many modules. Admittedly, I think this is an architectural problem rather than a doc problem, but it's going to be a long time before these sorts of architectural issues are resolved.

Developers are more likely to discover it in the repo than on elastic.co

I think this rationalization is flawed if not outright not true.

First, the docs are in the repo either way. If we stopped scattering our docs willy nilly throughout the repo and just put them in asciidoc, you could discover all of our development documentation directly in the repo right here. And GitHub renders asciidoc.

Second, nearly every large open source project hosts documentation on a website, and that is the primary way people consume those docs. Examples of this are everywhere. I think discoverability is precisely what gets lost when you abandon website-based docs in favor of docs exclusively in a git repo.

Content is formatted more nicely

I assume this means on GitHub compared to elastic.co/docs? I agree with that, though as I mentioned before, asciidoc is formatted on GitHub as well.

Content can be written in markdown which most developers are more familiar with

I agree that asciidoc is strange (it was foreign to me when I started working here), but it's a critical part of our job as engineers at Elastic, and that's true for all engineering teams across the entire company. I'm not saying we shouldn't embrace better tools when we can, but this one sounds a lot like "because it's not asciidoc", which isn't productive.

No need to build docs to confirm edits appear as expected (both for developer and reviewer). Keeping changes in the repo reduces context switching overhead. (added by Stacey)

Again, asciidocs is in the repo, so I'm not sure what that's about. Once you start writing enough asciidoc, you don't need to scrutinize every change you make on a built copy, just like when you write markdown you don't need to open it up in GitHub to see if looks OK. And the build step does really valuable stuff, like telling you when you've broken links, which is critical to maintaining comprehensive documentation. Markdown doesn't solve this problem, it just ignores the problem.

I think that there would be a compelling argument for keeping all of our docs on elastic.co if we officially supported third-party plugin development. But considering that we don't, I think it sends a confusing message to our users to officially publish docs to do things we officially don't support.

I'm not really sure what this is about, but we support third-party plugin development. That's why we have the plugin docs, why we list known plugins, why we try to help folks in our forums, why we try to blog about big plugin system changes like the es client stuff that happened with tribe, etc. I do think we could do a much better job supporting these developers, which is one of the reasons why I think this remove-docs-from-website proposal is a step backwards.

I do want to bring up a major frustration I have working with Kibana from a docs/dev perspective. The design team normally has to touch nearly every piece of Kibana which means we need to run it with everything on. I'd really love if we could maintain docs that described not just how to get local running, but how to get local running with all of the various systems running. Beyond that, it would be awesome to have some sort of one-liner that load everything up together

This is a great idea, and we can all benefit from thinking in such broad terms about the utility of our docs. But docs like this are precisely why trying to adhere to a "docs near code" thing can be problematic - what code would these docs live near? The answer is no code in particular, so instead we'll want to keep them in some clear directory. Perhaps a docs directory at the root of the repo?

[timroes]

I want to highlight another point about asciidoc, besides maybe unfamiliarity: It's just not powerful enough (neither would Markdown be) for all the needs around technical documentation. We can compile a lot of stuff (like JSDoc or Swagger specs) to asciidoc and compile them for the docs, but that doesn't anyhow means that this is a good solution or equivalent to proper documentation systems.

As far as I understood @clintongormley e.g. the Cloud API (like this example) is compiled from Swagger Spec. I don't think that output can anyhow be considered as useful as a proper Swagger UI (like the petshop example) or any other proper interactive tool.

The same is true for JSDoc in my opinion. Of course there are tools to convert JSDoc to markdown (and I think the same for asciidoc). Still I don't think that something like that markdown is anhow comparable to a proper interactive JSDoc page:

Thus I would highly prefer not having JSdoc or Swagger compiling down to Asciidoc in our documentation, due to the limitations and missing interactivity we would have from that. I neither think that we should put all of those in separate systems, since we don't want to split up tutorials, e.g. on how to embed visualizations from the actual API documentation on the visualiz loaders.

</my-two-cents>

[stacey-gammon]

@epixa,

Again, asciidocs is in the repo, so I'm not sure what that's about.

Yes, you are right, I meant more that you don't need an additional repo to build them (I could be wrong, but I think you also need to pull down the doc repo to properly build the docs).

Once you start writing enough asciidoc, you don't need to scrutinize every change you make on a built copy, just like when you write markdown you don't need to open it up in GitHub to see if looks OK.

I pretty much always open it up in github to see if it looks okay.

And the build step does really valuable stuff, like telling you when you've broken links, which is critical to maintaining comprehensive documentation.

So sometimes we can get away with not building them, but sounds like you are encouraging us, for the most part, so build them to catch these problems?

Markdown doesn't solve this problem, it just ignores the problem.

Yea, I acknowledge this.

The truth of the matter is, the more onerous we make it for developers to write documentation, the less documentation we will have. Given the amount of stuff on everyone's plate, sprucing up documentation has be extremely fast to warrant the time investment. Right or wrong, I closed my PR out when I saw there was already some asciidoc documentation for tests.

Regardless of where documentation should live, we clearly have a knowledge problem on the team about this - both CJ and I, in the same day, wrote up some functional test docs in markdown and didn't think to look in asciidoc for existing docs.

[cjcenizal]

we clearly have a knowledge problem on the team about this

I agree. If I ask myself why is there a knowledge problem, my own answer is that I simply don't think to look in our docs directory (or on elastic.co) to understand the code I'm looking at. I look at the code itself first, inline comments second, a block comment at the top of the file third, and some root-level documentation last. I'm not saying this is right or wrong, just describing my own natural tendencies. If other people have the same tendencies, then it makes sense to put docs where they'll be found.

We should decide on a long-term vision for our docs, and form a short-term plan which moves in that direction. I think Tim's idea to create a Kibana-specific dev docs site is terrific, and it sounds like something we should discuss during our roadmap planning. If we decide to work towards that goal then would moving all dev docs to asciidoc or markdown move us closer to it? I think that depends on how Tim's docs system would be implemented, i.e. would it consume a single docs directory, or would it comb through src and x-pack for docs?

On a side-note, @epixa is supporting third-party plugin development a recent change? As long as I've been on the team, my understanding has been that we've acknowledged that our APIs (term applied loosely since I believe plugins can access any public property of any module, right?) are subject to breaking changes without warning and because of this, we don't officially support third-party plugin development. My understanding has been that we don't give questions on the forums about plugin development the same level of support as product questions for this reason. I thought one of the goals of the new platform was to create our first set of stable and supportable APIs for plugin development. Have I misunderstood?

[havidarou]

I agree with @timroes that elastic.co maybe it's not the best place for the documentation, although pointing there to the repository's documentation (if that is where you end up doing it) would be really helpful.

Regardless of the place and format you decide for it, and given your high change rate, pointing out what is deprecated and at least a somewhat vague description of where you are heading to in some key aspects (dependencies, behavior changes, etc) can help to give developers a picture of what is and will go on.

Regards.

[epixa]

The source of my concern is that by moving docs off of elastic.co we'll lose visibility for important consumer facing documentation, or we'll encourage further neglect of already heavily neglected areas of consumer facing documentation. I think this concern is legitimate, but it doesn't mean we need to just publish everything that looks like a doc to elastic.co, and it doesn't mean that certain things published to elastic.co today have to continue to be.

We have a couple different types of documentation, and it's natural for different documentation methods to be appropriate depending on the purpose of the docs.

Product documentation is targeting consumers of Kibana, which include people deploying Kibana, using the UI, the REST API, or our plugin interfaces. It's important that this documentation be published to elastic.co for many reasons including discoverability. It's also important that it reside in the repo since it's tied to the overall experience of using Kibana, relevant to specific branches and even commits, etc.

As a company, we've embraced asciidoc for our product documentation, so it is the defacto standard for any documentation that gets published to elastic.co. That doesn't mean we can't introduce more effective ways to document certain things, especially for highly technical documentation like REST APIs and plugin interfaces, so I fully support the effort behind the proposals @timroes has made here. But in the absence of that alternative existing, we should write those docs in asciidoc.

Developer documentation is targeting people that contribute directly to the Kibana repo. This can be the contents of our contributing guide, style guides, jsdocs, etc. This documentation will take different forms, and it doesn't need to be published on elastic.co, though it can be published there. Some of these docs might not be published anywhere and will instead exist exclusively in the repo.

Putting a markdown file at the root of a module to describe it in more detail is great. This is basically just a comprehensive comment in the code as far as I'm concerned. We do this in a couple places already.

Those examples are documenting the modules themselves. It's not documenting how you consume those modules through yarn kbn or the functional test clis because that is only a logical place to document those behaviors if you are already familiar with the implementation details of those commands, which means they are not inherently discoverable.

Things like developer CLI commands need to be documented at a centralized or root level, like we do with the contributing guide today.

There is going to be overlap here, and that's OK. For example, the i18n module readme is really comprehensive, and it includes information that is important for plugins authors to have, like how you are suppose to write your templates so that they can be translatable. It also includes information that is only relevant if you're contributing to the kbn-i18n module in the Kibana repo. When we finish rolling out i18n across Kibana and start productizing it, we need to either duplicate or pull out the information that's relevant for plugin developers into consumer facing documentation. It certainly can be pulled out sooner than that as well.

[cjcenizal]

Thanks @epixa. By these criteria, would you consider the elastic.co examples in the issue description to be developer docs? Would they be better located as markdown files within the repo? Would functional testing guidelines be developer docs, and would they belong in the repo?

[epixa]

@cjcenizal Some of them are only relevant to people writing code for the Kibana repo, but not all of them. For example, Managing Dependencies is only a concern when you're contributing to the kibana repo, however Developing Visualizations is important for plugin developers.

If we're moving core dev stuff into markdown files in the repo, we can probably move some things around and remove the "core development" section entirely from the elastic.co docs.

On the testing front, we intentionally provide Functional Tests for Plugins documentation for plugin authors to run their own selenium test with Kibana's, which should continue to be published to elastic.co. Guidelines for how we as a team want to write tests in the kibana repo is a different story, and I think that's what you're referring to with #9162. That could just be a markdown file in Kibana.

[timroes]

@epixa Exactly because of what you stated in your previous comment, I would not separate only between Product documentation and Developer documentation, since I think we actually have to handle three target audiences:

Target audiences

Product documentation

Everyone who uses Kibana. These users usually will always work against a build Kibana version and never "see" any source code of it.

This should hold all the description on how to install and use Kibana (the UI). I also think there is no discussion that this should be on elastic.co in the docs.

Plugin developers

Everyone who want to develop a plugin against Kibana. Those people usually will work with a source code variant of Kibana locally, need a web development environment (and thus have different network requirements) and most likely need a deeper technical knowledge.

Developer documentation

People who want to actually modify existing Kibana source code. These effect also documenting internals of our systems, e.g. how does the visualization rendering pipeline work internally. In the future there might be less "internal APIs", due to the new platform and proper separation of APIs, but there will still always be internal documentation to plugins, like above mentioned rendering pipeline - which should not be modified by any plugin, but just consumed via its APIs - which should be documented for the "Plugin developers" audience.

I think mixing Product documentation and Plugin developers, as @epixa suggested, is not a good solution for us in the long run, for a couple of reasons:

• It's two different target audiences. Usually only a very low percentage of all users who would consume "Product documentation" will fall into the "plugin developer" group and would want to read this.
• Thus putting this highly technical documentation into the same system, might alienate users from the first group ("That's just a developer tool."). This will be more and more of an issue the more we are moving into less technical users, and more business analytics users.
• We have different technical (and maybe design) needs for those systems. We will never need JSDocs in the "Product documentation", but we are in high need for it for the "Plugin Developers" documentation.

I think we are going down a very dangerous part mixing those two first groups together, for both of the groups: We'll alienate the first group, and don't give the second group the proper documentation (currently mostly because of technical limitations, e.g. JSDoc) that they require.

I would highly suggest we are not trying to mix those groups in the future.

Places to document

I think there is no discussion needed, that "Product documentation" will always stay on the elastic.co side.

For several reasons, that I already mentioned in my previous and this comment, I would really like to see "Plugin Developer documentation" to live in a different place: different technical requirements (proper JSDoc), not alienating both groups via mixed docs, (not mentioned till now): I think elastic.co has a very product focusing design, which is good for marketing purposes, but horrible to put developer documentation into.

For the "Developer documentation" (for core development) I am not having a strong opinion right now, I think it could either reside in a system similar to the "Plugin developer documentation" - or within the same system, if that system could clearly separate "internal" documentation from "plugin documentation". Maybe it's also fine to have that documentation just within the repository as markdown - I think that's what most people voted for in this discussion.

A note on REST APIs

I haven't mentioned REST APIs in any of the above, since I am not sure if they belong to the first or second group. There are reasons for both imho:

• REST APIs can be consumed on an regular installed instance, i.e. making it rather product documentation.
• REST APIs usually will be consumed by more technical savvy people looking into automation, i.e. making them perhaps a better fir for plugin developer documentation.

No matter where they should go, that system should imho be able to do way better, than rendering Swagger API to asciidoc and output in in such a format.

Note on the comunity

I think it's also important to hear more opinion on that topic from developers (2nd of 3rd category) from the community. @havidarou thanks for your input already. I also saw, that @trevan voted for markdown, so in case you have any additional thoughts you want to share, I would highly appreciate your input here, too.

[epixa]

I think we are going down a very dangerous part mixing those two first groups together, for both of the groups: We'll alienate the first group, and don't give the second group the proper documentation (currently mostly because of technical limitations, e.g. JSDoc) that they require.

This seems over the top and a misrepresentation of what I proposed.

Having both developer documentation and product documentation on elastic.co doesn't alienate folks - we've had docs on contributing to Kibana in our own docs since 5.0, we also have developer-facing docs in Elasticsearch, Logstash, and Beats. This is nothing new and while there are legitimate concerns about whether we're using the right format/tools for developer docs, it's mere existence together with the product documentation has never been an issue.

I agree completely that technical documentation (regardless of whether it's targeted at contributors or plugin developers) can and should be published with different audiences in mind, which includes using different tools where appropriate. I thought I made this clear when I said:

That doesn't mean we can't introduce more effective ways to document certain things, especially for highly technical documentation like REST APIs and plugin interfaces, so I fully support the effort behind the proposals @timroes has made here.

The only qualifier I added to that was that coming up with a new way to publish docs doesn't happen overnight, so in the meantime we need to use the tools at our disposal.

To be honest, I don't really understand your response, Tim. I feel like that response only makes sense if I'm proposing that we keep our developer facing documentation exactly as it is today, which I'm not now and never have before. Within Kibana, our technical documentation exists in the form it does today because no one has proposed and implemented a better alternative. Your idea around technical documentation that you showed to me a couple weeks ago seems like an awesome alternative, but it needs to be turned into a definitive plan and then the execution needs to be staffed.

[clintongormley]

I completely agree with @epixa's comments in #21269 (comment)

I just don't buy this idea that REST APIs can't be documented by elastic.co, and that you have to have a dynamic website to read them on. I call bullshit here. Elasticsearch has successfully done this for years, and it has a massive REST API! Some people moan about the docs, and some people have praised how good they are.

This is purely personal preference. Well my personal preference is different. Swagger's own docs have long pages with a nav bar. Just like elastic.co. So what actual changes would you make to our current system to make it objectively better?

Perhaps you don't like the styling of the docs pages? Fair enough, IANAD. Feel free to contribute improvements, that then benefit all of the elastic.co docs.

Again, asciidocs is in the repo, so I'm not sure what that's about.

Yes, you are right, I meant more that you don't need an additional repo to build them (I could be wrong, but I think you also need to pull down the doc repo to properly build the docs).

You need to install a tool to parse Asciidoc??? How is that a barrier? We can't use javascript anymore because we need to install node???

Asciidoc is so similar to markdown (except more powerful, with features that allow us to keep our docs in a good state), that I don't get what the problem is here. People are happy to learn Typescript, but coping with a docs markup language is too much to ask?

I am against is coming up with a whole new system for docs for Kibana. I know how much work it takes to build and maintain such a system. Let's be specific about the problems with our current docs and figure out how to make those better.

[cjcenizal]

I'm closing this issue as the original question about where core-contributor-facing development docs belong. Anything developer docs that don't directly aid plugin development will be moved to or created as markdown files in the Kibana repo.

Thanks for participating everyone!