Add gRPC API and shell command

[adambabik]

This change adds a new command server which starts a gRPC server with two services:

Install buf and grpcurl first. Both can be found in brew.

$ grpcurl -protoset <(buf build -o -) list
runme.kernel.v1.KernelService
runme.parser.v1.ParserService

The first allows creating kernels - long running processes which you can interact with using the service.

The second allows to serialize and deserialize Markdown content.

Run

Remember to rebuild runme using make and then execute:

$ ./runme server --address ./runme.sock

Deserialize

data=$(echo -n '# Hi' | openssl base64) 
grpcurl -protoset <(buf build -o -) \
  -d "{\"source\": \"$data\"}" \
  -plaintext -unix ./runme.sock \
  runme.parser.v1.ParserService/Deserialize

It should result in:

{
  "notebook": {
    "cells": [
      {
        "kind": "CELL_KIND_MARKUP",
        "value": "# Hi"
      }
    ]
  }
}

Kernel

If you have jq, you can pipe the result of grpcurl to decode the output automatically using | jq -r '.stdout' | base64 -D.

First, a session needs to be created:

$ grpcurl -protoset <(buf build -o -) -d '{"command_name": "/bin/bash", "prompt": "bash-3.2$"}' -plaintext -unix ./runme.sock runme.kernel.v1.KernelService/PostSession

The output will contain a session ID:

{
  "session": {
    "id": "cerbqj43c37nq95oe3ng"
  }
}

Next, you can execute commands in that session:

$ grpcurl -protoset <(buf build -o -) -d '{"session_id": "cerbqj43c37nq95oe3ng", "command": "sleep 1\necho Hello\n"}' -plaintext -unix ./runme.sock runme.kernel.v1.KernelService/Execute

And you should see a stream of outputs (they are base64-encoded so you need to use something like echo -n $data | base64 -D):

{
  "stdout": "c2xlZXAgMQ0K"
}
{
  "exitCode": 0,
  "stdout": "ZWNobyBIZWxsbw0KSGVsbG8NCg=="
}

The last message is identified by presence of exitCode.

Protobuf files and code generation

This change contains generated code for Protobuf files in Go and TypeScript. For TypeScript, check out Protobuf-ES which is used to generate TS code. However, please wait for confirmation how to consume proto files and generated stubs as we will likely try to use Buf Schema Registry.

[sourishkrout]

How would you feel about renaming daemon to kernel to be more in line with the notebook architecture terminology @adambabik?

[sourishkrout]

👍 very much like the direction this is going. I've got a good amount of experience with GRPC services in Go, Python, TS/JS. Curious to try out Buf to simply some of the Grpc/Protobuf integration woes.

Filed this PR with the example I ran locally: #94

[adambabik]

@sourishkrout it's ready for testing. I tried to document how to do it in the PR description but if something is unclear let me know.

It's still very early, especially if one has a rich multiline prompt, then detecting it might not work. For that issue, I consider another approach which is setting prompt ahead of time.

[sourishkrout]

Played through the examples. Works well. I did see it continuously streaming when I ran npx dancing-bear which is the example I used in the latest blog post. Staying put to get confirmation on how to integrate via TS and Buf's registry. 👍

[sourishkrout]

Read over the code and this looks like a good baseline to me 👍.