SDKs Documentation in API References - MVP

[curquiza]

Currently, there is no SDK documentation except the READMEs and the Getting Started in the docs.

But, in this part of the docs, the code is hard-written: the code snippets are not automatically generated. If the SDK methods change, the documentation needs to be changed too.

We should find a way to link the docs repository and each of SDKs.
In one PR, the SDK maintainers should be able to change the code of the SDK and the docs at the same time.

With this kind of solution, it would be able to add more code snippets in API References.

Related issue

How?

A "simple" solution would be to add a template file in each SDK that the documentation would be able to read and then generate code snippets according to this file.

Example of a template file:

###
# guides/introduction/quick_start_guide.html#create-your-index
$ gem install meilisearch
---
require 'meilisearch'

client = MeiliSearch::Client.new('http://127.0.0.1:7700')
index = client.create_index('movies')
###

###
# guides/introduction/quick_start_guide.html#add-documents
require 'json'

movies_json = File.read('movies.json')
movies = JSON.parse(movies_json)
index.add_documents(movies)
###

###
# guides/introduction/quick_start_guide.html#search
index.search('botman')
###

Steps

• Find a great syntax for the template files: thinking about the delimiters, the "title" of the code snippet...
• In the docs part: create a parser able to read the template and then, generate the code snippets => most of the work is in the docs part.
• Find a way to regenerate the docs if a template changes (there is scheduler with GHA to check them on regular bases)
• The docs should provide a script that generates an empty template.

Bonus:

• If the template changes, a bot should create a PR (or at least an issue) to change the current template for each SDK repository.

Bonus ++:

• Find a way to test the lines of code in the template files, to be sure that we provide good examples in the docs.

[bidoubiwa]

SDK code samples in documentation

State

As of today, only the getting started guide has code samples of most of the SDK's.

Objective

Our objective is to get every example in the documentation to have different tabs with the different SDK's like this:

Now to achieve this results those are the steps:

• Creating the templates
• Filling the samples
• Collecting the samples
• Showcasing the samples in tabs

Creating the templates

Because it is more easy to write code in a file that has a good extension (.js for javascript) I made a little CLI that generated the files based on a list of samples:

( for this example I only took the list of examples for the API references of the indexes route)

For this section to be completed:

• review CLI
• Finish the list of all samples list that helps generate the files

Filling the samples

In each generated file I have put the link to the place in the documentation where the sample will be. This is to help find a context to each example.

(link must be improved, they are bad for example purpose)

• Better links to documentation
• Complete the samples in every language

Collecting the samples

Once every sample has been completed, it would be a hassle for the documentation to require each sample one by one.

I made this little tool that compiles every sample into a json file. This is the file that will be gathered by the documentation:
(i have put the same code in each sample file just for the example)

As you can see this example generated a php-sample.json. This sample contains the following:

[
  {
    "id": "create_index_examples_1",
    "content": "<?php\n    $index = $client->createIndex('books');\n?>"
  },
  {
    "id": "delete_index_examples_1",
    "content": "<?php\n    $index->deleteIndex();\n?>"
  },
 ...
]

This file will be fetched on build in the documentation. Because it is in JSON file, no parsing is required at all by the documentation to read them as JSON objects.

• Have the compiled file language-sample.json at the root of every SDK repository. Or in an another repo.
• Script to fetch all files in the different SDK's on build

Showcasing the samples in tabs

In the documentation, we will use the following component to generate the tabs with all the fetched content:

<Codesample id="create_index_examples_1" />

This will add at that position a tab element with all the fetched SDK's.

• Vue Component that will make the tabs with the different language samples

Bonus

To make it more easy for a user of a specific SDK to see all the examples in a given language this is what stripe suggest in the documentation :

• Link based: https://stripe.com/docs/api?lang=go
  Every code sample is now by default on go
• Clickable choice:

By clicking on one of this logo it will change all the default samples to the one chosen.

[tpayet]

• We have approximately 40 method to cover.

• We have methods in the wrapper that are not in the original API, how do we cover them?

[bidoubiwa]

We have methods in the wrapper that are not in the original API, how do we cover them?

@tpayet Could we discuss this here : #15

[eskombro]

In my opinion, the proposition made by @bidoubiwa makes sense, thanks! First of all, this will only work if it is really easy to identify which is the file example that you have to modify when you change a method in a SDK.

I am not sure about some details:

1. CLI

• PROBLEM:
  The need to install a CLI for creating the examples. It seems simple, but is one additional step and tool, and adds complexity for contributions, we don't want to discourage contributions.
• POSSIBLE SOLUTIONS:
  Have a shell(sh/bash?) script that generates the files, hosted in the docs repo, and that can be called with (for example) curl https://github.com/meilisearch/documentation/examples/generate-examples.sh | sh. We can also have a script in the root of each repo, it would make a simple call to that main script from the docs. In that case, we don't need to maintain any script in the repos, because it is calling an external script, but when you clone the repo you can just run sh create-exaples.sh in the root and you will have your files.

2. Compiling examples

• PROBLEM:
  Adding additional steps for contibuting (same as CLI)
• POSSIBLE SOLUTIONS:
  As we discussed, it might be better to do the compilation of the examples in the DOCS and not in the SDK. In this way, the developer who wants to contribute just need to clone the repo, run the script if he needs to generate examples and modify the files he needs to modify (which shouldn't be very often.

3. Two different files, one for the example,one for the response

• PROBLEM:
  Adding unnecesary complexity
• POSSIBLE SOLUTIONS:
  Integrate response and code example in the same template

[tpayet]

What we are trying to achieve:

• SDK's examples in the API Reference part of the documentation
• There is no tool to automate this, so we will try to build a workflow that ask for the minimum of work from the SDKs contributors and the minimum of work for us to automate the job.

Real-time update: Since VuePress is built as a static website, we may not be able to use Vue as a cliend-site rendering which is sad because we could have imagined the Vue app to get on client-side the examples in each SDK GitHub repository.
So we will have update on SDKs examples, every time a build is done. Am I right?

We should have a VuePress component that will get the SDKs' examples on each GitHub repository at build time. If no example is available, the component won't build the tab associated with that language.

We can have one or multiple file in each repository that should be hidden

./meilisearch-php
  | scripts/
  | src/
  | tests/
  | .documentation-samples #hidden file or directory
  | ...

This way the VuePress component can get each sdks sample based on this url: https://github.com/meilisearch/meilisearch-<language>/blob/master/.documentation-samples

Using the described workflow, we only have one component to write and no github actions do setup or CLI to install

Documentation sample

How to format this document or this repository?

I would prefer a simple document to ease the work on the component side and because we only have around 40 examples to write down for each sdk. This is something that could evolve later ofc, but we can start with a simple file.
We should have a slug or "human readable id" to link code samples in each SDKs to the right documentation snippet tab.

E.g. using YAML for examples in Ruby:

./documentation-samples
---
create-your-index: |-
  require 'meilisearch'

  client = MeiliSearch::Client.new('http://127.0.0.1:7700')
  index = client.create_index('movies')

add-documents: |-
  require 'json'

  movies_json = File.read('movies.json')
  movies = JSON.parse(movies_json)
  index.add_documents(movies)

What format?

• JSON: difficult to write multiple line with indentation
• YAML: difficult to manage indentation but we can have easily multiple lines
• CUSTOM: Easy to parse, but we have to make our own parser

In Rust documentation examples are compiled to ensure that examples are working. In Ruby, you can use your test to generate documentation (so your documentation is tested). We can imagine a future where we introduce special comments in our tests suite to get examples from our test suite in order to ensure that every example is always working.

Conclusion

We may not have something very modular here but I think this solution would provide us with a working solution ASAP since we only have to agree on a format and write the VuePress component that will get each sample based on convention we can agree on. We are avoiding here CLIs tools or github-action or even transpilation scripts ^^
I get that this solution is not the best for the sdk developer since there is a YAML/chooseyourformat but I think it is a compromise against the simplicity of deployment.

[curquiza]

Why not using markdown as a format?

• Everybody knows that markdown is for docs, not for code execution
• The is no highlight or delimiters issue thanks to ``` delimiters.
• The delimiters in marknow are explicit: the SDK maintainer knows if he/she has to write a bash example or or a Python example.

[tpayet]

Markdown is a markup language, not a data-serialization format so we won't have slug or ids natively supported but we can work around something for sure

[eskombro]

I really like your idea @tpayet !

The format/language discussion is indeed an important one for this implementation. Doing our own format will for sure create a lot of work not only now, but also in the future, as it evolves. YAML and MarkDown (with a slug/id workarround) seem like the most promising solutions for that! In fact, the md solution is great because it would be easy for the developer to rely on other tools for highligthing etc, and be sure about the result he will have in the published documentation.

[bidoubiwa]

After reading through all of your suggestions and after a talk with @curquiza this is the final proposition

Foreword: We decided to go for stripes solution by not adding the response example for each SDK. This example will remain the json one. As it is comprehensible what it will returned no matter the language.

Steps required

• Get the template
• Template format
• Samples format
• Fetch the samples
• Display the samples

In order to get everyone's votes, let me propose the following template

- Get the template: 1
- Template format: 3
- Fetch the samples: 2
- Display the samples: 1

Feel free to add your suggestions and/or additional information. I can always edit this post.

Template format

The templates are generated based on a file that contains all places in the documentation that requires an example.

Solution 1: one file solution

YAML file generated this way:

./documentation-samples

import_sdk_1: |-
    
get_index_request_1: |-

get_indexes_request_1: |-

create_index_request_1: |-

delete_index_request_1: |-

get_document_request_1: |-
...

How it is expected to work ?

Yaml accepts multi-line "strings", nevertheless, it is expected that each sample has the appropriate number of tabs to be valid YAML.

(some random example of single, empty and multiline examples)

import_sdk_1: |-
    import MeiliSearch from MeiliSearch
get_index_request_1: |-
    require 'meilisearch'

    client = MeiliSearch::Client.new('http://127.0.0.1:7700')
    index = client.create_index('movies')
get_indexes_request_1: |-
create_index_request_1: |-
    client.createIndex("movies")

This template should be added at the root of each SDK

./meilisearch-php
  | scripts/
  | src/
  | tests/
  | .documentation-samples #hidden file or directory
  | ...

Solution 2: multifile solution

In a folder called .samples/ at the root of each SDK

./meilisearch-php
  | scripts/
  | src/
  | tests/
  | .documentation-samples/ #hidden file or directory
       | get-index-request-1
       | get-indexes-request-1
       | create-index-request-1
       | delete-index-request-1
       | ...
  | ...

Each file is empty and expect the sample in the appropriate language.

How it is expected to work ?
samples/get-index-request-1 should contain the following:

client.getIndex('movies')

Samples format

How do we right examples for the documentation ?

Solution 1: no-complexity no-index

client.getIndex('movies').addDocuments(docs)

Solution 2: no-complexity index

index.addDocuments(docs)

Solution 3: some-complexity index

json_file = open('movies.json')
movies = json.load(json_file)
index.add_documents(movies)

Solution 4: all-complexity index

index = client.get_index('movies')
json_file = open('movies.json')
movies = json.load(json_file)
index.add_documents(movies)

Imports can be added to any of the above suggestion, feel free to precise your vote with imports.
exemple in ruby :

require 'meilisearch'

client = MeiliSearch::Client.new('http://127.0.0.1:7700')
index = client.create_index('movies')

Get the template

The first step is to get a template in each SDK that the contributor will use to complete all code samples. This make it easier for him because it does not require to search it up, and this gives us the possibility to add human readable slugs.

Solution 1: Documentation script

Generate the file(s) from the documentation
yarn docs:generate:template

Solution 2: HTTP request

From a HTTP request
curl docs.meilisearch.com/samples

Fetch the samples

Solution 1: server-side on build

Server side on build.

While the documentation is building, from a list of SDK's, it will fetch all samples files (multifile or singlefile) from every given repository.

This means, that the changes made to the different samples files will only be published once the documentation is built and published again. A Possible solution to that would be to have a scheduler CI that checks on changes every now and then (2 hours suggested by clementine).

Solution 2: Client side on load

It is in fact possible to gather all the code examples at runtime. Using the state library vuex to make the samples accessible everywhere. It requires the user to download all sample files (easier with one-file suggestion) every time he comes to the documentation. Which is maybe less heavy than I think.
This means that we have no outdated content but the development time is a little longer than solution 1 as I have to understand how the state manager works in vue.

Display the samples

Only one solution here :

Solution 1: Vue component

Common to both solutions:

The CodeSample vue component is used to display the right samples at the right position in the documentation.
<CodeSample id="get-index-request-1" />

Based on the repository from which each sample originate the code will be displayed this way:

\``` [language]
  <sample>
\```

Unless specified otherwise by the configuration file in the documentation that can overwrite it with another present (example bash instead of the given language).

[eskombro]

Final decision for MVP (team meeting):

• Template format: 1 (one file solution). YAML, single line
• Samples format: 1 (no-complexity no-index)
• Get the template: Template will be available in the docs repo for now
• Fetch the samples: 1 (server side)

Edit from Clementine:

• Fetch the samples: 1 (server side)

Not in the MVP if I'm not wrong. We will re-run the CI for the build if needed. But, yes, it will be in server side later as a future improvement.

[curquiza]

The MVP was merged in the docs!! -> meilisearch/documentation#356
Still need to complete the yaml files in different SDKs. All the issues are open and well-detailed thanks to @bidoubiwa.

I'm proud to close this issue 😌