WiP - GoCSI

[akutz]

This patch provides a Go-based CSI client and server capable of supporting additional storage platforms at runtime via Go plug-ins.

Note: For anyone reviewing this, please do not comment on the vendoring of grpc or protobuf under the mod/moc directory. I will be updating this PR to explain why vendoring of these packages is necessary. The tl;dr is vendoring+Go plug-ins+gRPC type registry.

csc

The Container Storage Client (csc) is a CLI program written in Go that provides one-to-one mappings with the currently available CSI RPCs.

$ csc
usage: csc RPC [ARGS...]

       CONTROLLER RPCs
         createvolume (new, create)
         deletevolume (d, rm, del)
         controllerpublishvolume (att, attach)
         controllerunpublishvolume (det, detach)
         validatevolumecapabilities (v, validate)
         listvolumes (l, ls, list)
         getcapacity (getc, capacity)
         controllergetcapabilities (cget)

       IDENTITY RPCs
         getsupportedversions (gets)
         getplugininfo (getp)

       NODE RPCs
         nodepublishvolume (mnt, mount)
         nodeunpublishvolume (umount, unmount)
         getnodeid (id, getn, nodeid)
         probenode (p, probe)
         nodegetcapabilities (n, node)

Use the -? flag with an RPC for additional help.

$ csc ls -endpoint tcp://127.0.0.1:8080
id=1	name=Mock Volume 1	
id=2	name=Mock Volume 2	
id=3	name=Mock Volume 3

csd

The Container Storage Daemon (csd) is a CLI program written in Go that provides gRPC endpoints for all of the currently defined CSI services: Controller, Identity, and Node. The program's storage support can be extended at runtime via Go plug-ins.

$ CSI_PLUGINS=$(pwd)/mock.so csd mock
2017/06/26 02:06:34 loaded plug-in: mock.so
2017/06/26 02:06:34 registered endpoint: mock
2017/06/26 02:06:34 mock.Serve
2017/06/26 02:07:06 csd.ListVolumes
2017/06/26 02:07:06 ...Volume.ID=1
2017/06/26 02:07:06 ...Volume.ID=2
2017/06/26 02:07:06 ...Volume.ID=3

csp

This package contains packages that can be compiled as stand-alone CSI daemons (similar to csd) as well as Go plug-ins that can be loaded into csd in order to extend the latter's storage platform support at runtime.

$ go run csp/moc/main.go
2017/06/26 02:08:51 mock.Init
2017/06/26 02:08:51 mock.Serve
2017/06/26 02:08:53 ...Volume.ID=1
2017/06/26 02:08:53 ...Volume.ID=2
2017/06/26 02:08:53 ...Volume.ID=3

Travis-CI

The forked repo from which this PR was generated is configured to use Travis-CI. The current builds of this PR's branch show the automated testing using the produced artifacts:

$ make -C gocsi test
make: Entering directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi'
make -C csc build; make -C csd build; make -C mod build;
make[1]: Entering directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/csc'
make[1]: Nothing to be done for `build'.
make[1]: Leaving directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/csc'
make[1]: Entering directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/csd'
make[1]: Nothing to be done for `build'.
make[1]: Leaving directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/csd'
make[1]: Entering directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/mod'
make -C moc build;
make[2]: Entering directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/mod/moc'
make[2]: Nothing to be done for `build'.
make[2]: Leaving directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/mod/moc'
make[1]: Leaving directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/mod'
CSI_PLUGINS=$(pwd)/mod/moc/moc-csi-plugin.so csd/csd mock > csd.log 2>&1 &
sleep 2s
csc/csc ls
id=1	name=Mock Volume 1	
id=2	name=Mock Volume 2	
id=3	name=Mock Volume 3	
csc/csc new -o norootsquash,uid=500,gid=500 \
                -t ext4 -requiredBytes 107374182400 \
                -params color=purple,up=down \
                "My New Volume"
id=4	name=My New Volume	
csc/csc ls
id=1	name=Mock Volume 1	
id=2	name=Mock Volume 2	
id=3	name=Mock Volume 3	
id=4	name=My New Volume	
kill $(ps aux | grep '[c]sd' | awk '{print $2}')
cat csd.log
2017/06/26 07:00:05 loaded plug-in: /home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi/mod/moc/moc-csi-plugin.so
2017/06/26 07:00:05 registered endpoint: mock
2017/06/26 07:00:05 mock.Serve
2017/06/26 07:00:07 csd.ListVolumes
2017/06/26 07:00:07 ...Volume.ID=1
2017/06/26 07:00:07 ...Volume.ID=2
2017/06/26 07:00:07 ...Volume.ID=3
2017/06/26 07:00:07 csd.CreateVolume
2017/06/26 07:00:07 CreateVolume.CapacityRange=required_bytes:107374182400 
2017/06/26 07:00:07 CreateVolume.Name=My New Volume
2017/06/26 07:00:07 CreateVolume.Parameters=map[color:purple up:down]
2017/06/26 07:00:07 CreateVolume.VolumeCapabilities=[mount:<fs_type:"ext4" mount_flags:"norootsquash" mount_flags:"uid=500" mount_flags:"gid=500" > ]
2017/06/26 07:00:07 ...Volume.ID=4
2017/06/26 07:00:07 csd.ListVolumes
2017/06/26 07:00:07 ...Volume.ID=1
2017/06/26 07:00:07 ...Volume.ID=2
2017/06/26 07:00:07 ...Volume.ID=3
2017/06/26 07:00:07 ...Volume.ID=4
received signal: terminated: shutting down
server stopped gracefully
make: Leaving directory `/home/travis/gopath/src/github.com/container-storage-interface/examples/gocsi'

[carmark]

How about separating the commit into two commits? one is real patch and the other is vendor code.

[akutz]

Hi Lei, Because it wouldn't compile that way. The decision is more straight-forward than that. Should it build with a "go ..." sans any "make" beforehand? If the answer is yes then the vendor code should be part of the commit. IMO a PR should be a single, atomic commit in order to to wholly add a feature in its entirety or remove it completely later. I see no value on dividing it into multiple commits at the point of merging it. Especially not when one depends on the other, and in this case N would depend on Commit N+1, which just doesn't make sense.

…

-- -a

On Jun 26, 2017, at 8:24 PM, Lei Xue ***@***.***> wrote: How about separating the commit into two commits? one is real patch and the other is vendor code. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[akutz]

To be clear, I'd opt for removing the vendored code entirely and just declaring that the Makefiles are required for building or installing anything.

Which is what I do on my projects. However, for an example I wanted things to be cleaner and more aligned with go's normal tooling workflow.

[akutz]

Hi @carmark / @jdef / @edisonxiang,

Please see the updated README for additional information. I've included the current version of it here for your convenience:

---

gocsi

This project provides a Go-based CSI client and server capable of supporting additional storage platforms at runtime via Go plug-ins.

Name	Description
csc	A command line interface (CLI) tool that provides analogues for all of the CSI RPCs.
csd	A CLI tool that serves the CSI services -- Controller, Identity, and Node -- on a TCP port or UNIX socket. The storage platform support for csd is provided via Go plug-ins, loadable at runtime.
csp	A directory containing Go packages that can be built as stand-alone CLI programs capable of serving the CSI services via a TCP port or UNIX socket, similar to csd. Additionally, the csp packages can also be built as Go plug-ins that the csd program can load, extending its storage provider support dynamically, at runtime.

Getting Started

The example below illustrates how the above tools and Go plug-ins work together to provide a cohesive experience in line with the CSI specification.

Please note that the mock csp is used to fulfill the role of a storage platform. Also, because Go plug-ins are demonstrated this example must be executed on a Linux platform as Go plug-ins are not yet supported by other operating systems (OS).

# get and install the sources
$ go get github.com/container-storage-interface/examples/gocsi

# build the container storage client
$ go install github.com/container-storage-interface/examples/gocsi/csc

# build the container storage daemon
$ go install github.com/container-storage-interface/examples/gocsi/csd

# build the mock container storage provider
$ go build -o mock.so -buildmode plugin github.com/container-storage-interface/examples/gocsi/csp/moc

# export the CSI endpoint
$ export CSI_ENDPOINT=tcp://127.0.0.1:8080

# start the server (assuming $GOPATH/bin is in the PATH)
$ CSI_PLUGINS=$(pwd)/mock.so csd mock > csd.log 2>&1 &
[1] 19050

# use the client to ask for a list of volumes
$ csc ls
id=1	name=Mock Volume 1	
id=2	name=Mock Volume 2	
id=3	name=Mock Volume 3

# create a new volume
$ csc new "My New Volume"
id=4	name=My New Volume

# query the volume list again
$ csc ls
id=1	name=Mock Volume 1	
id=2	name=Mock Volume 2	
id=3	name=Mock Volume 3	
id=4	name=My New Volume

# kill the server
kill -HUP $(ps aux | grep '[c]sd' | awk '{print $2}')

# view the server log
$ cat csd.log
2017/06/26 01:54:48 loaded plug-in: mock.so
2017/06/26 01:54:48 registered endpoint: mock
2017/06/26 01:54:48 mock.Serve
2017/06/26 01:55:36 csd.ListVolumes
2017/06/26 01:55:36 ...Volume.ID=1
2017/06/26 01:55:36 ...Volume.ID=2
2017/06/26 01:55:36 ...Volume.ID=3
2017/06/26 01:55:47 csd.CreateVolume
2017/06/26 01:55:47 CreateVolume.CapacityRange=<nil>
2017/06/26 01:55:47 CreateVolume.Name=My New Volume
2017/06/26 01:55:47 CreateVolume.Parameters=map[]
2017/06/26 01:55:47 CreateVolume.VolumeCapabilities=[]
2017/06/26 01:55:47 ...Volume.ID=4
2017/06/26 01:56:04 csd.ListVolumes
2017/06/26 01:56:04 ...Volume.ID=1
2017/06/26 01:56:04 ...Volume.ID=2
2017/06/26 01:56:04 ...Volume.ID=3
2017/06/26 01:56:04 ...Volume.ID=4
received signal: terminated: shutting down
server stopped gracefully

Build Reference

GoCSI can be built entirely using go build and go install. However, Make is also used in order to create a more deterministic and portable build process.

Make Target	Description
build	Builds csc, csd, and all of the csps. Please note that this target does not install anything to the $GOPATH. Instead each of the packages' own Makefiles ensure that all code is built in a directory named .build inside the package using the go tool's -pkgdir flag in conjunction with the environment variable GOBIN. This ensures builds that are easy to clean.
clean	Removes the artifacts produced by the build target.
clobber	Depends on clean. Removes all generated sources and vendored dependencies.
test	Depends on build. Executes an end-to-end example using csc, csd, and the mock csp.
bencmark	Usses go test to illustrate the performance of the PipeConn used by csd to host the csps as in-memory CSI endpoints.
goget	Will scan the packages and execute go get for any missing dependencies.

Frequently Asked Questions

This section answers some of the common inquiries related to this project.

What is the purpose of GoCSI?

The purpose of GoCSI is not to provide a reference library or set of tools meant to be used by other, third-party or external consumers wishing to jumpstart CSI development. GoCSI is simply a series of small programs and packages that provide working examples and demonstrations of the CSI specification. If at some point the CSI working group wishes to build a set of reference implementations based on the contents of this project, then that is fine. If GoCSI stands forever as simply a series of examples, that's fine as well.

Why vendor Proto and gRPC in the csps?

There are two answers to this question. The first is simply so that the examples included in GoCSI can be built/executed with standard Go patterns without the need for an external build system such as Make or vendoring solution like Glide. The fact is that executing make clobber at the root of the GoCSI project will remove all of the vendored code as well as generated sources. Running make again will restore all of the removed code. It's simply by design to leave that code as part of the commit in order to make it as easy as possible to use the examples included in GoCSI.

The second answer is the result of weeks of research and trial and error. The short answer has to do with how Go manages packages, plug-ins, and the gRPC type registry. The long answer is summarized by golang/go#20481.

The purpose of the csps is to demonstrate the triviality of creating a package that can be built both as a stand-alone CSI endpoint and a Go plug-in, capable of providing support for a new storage platform to a separate, completely unrelated host program. Therefore the csps should have absolutely no relationship to any other GoCSI project, even csd, the program capable of loading the csps as Go plug-ins.

Truthfully the matter is much more detailed, but it won't be repeated here. For that please see the aforementioned Golang issue. Suffice it to say, the only way to create Go plug-ins that are useful in production software is to 1) vendor everything inside the plug-in's package and 2) use only Go stdlib types and empty interfaces when communicating with the plug-in from the host program.

Why generate Go sources from the CSI protobuf in multiple locations?

The answer to this question is similar to the answer of the previous question. If the csps referenced the csi package at the root of the GoCSI project then the csd program would not be able to load the Go plug-ins due to 1) duplicate gRPC type registrations and 2) a possibly different hash for the referenced package since the plug-ins are built separately and thus potentially from different sources.

[thockin]

I'm not sure I understand the value of the plugins mechanism here. IMO, the value of an example is the fact that it is approachable and digestible, and I worry that this is much closer to a real implementation than an example. Inclusion of EBS, for example, is a red flag.

I'd much prefer that, if we have examples, they are approximately toys with no cleverness beyond JUST what is needed to demonstrate fulfillment of the API.

I don't think an omnibus plugin is a bad thing, per se, but it a) needs bounds (e.g. only open standards) and b) is probably not an "example".

[akutz]

Hi @thockin,

I'm inclined to agree with you. I originally targeted the libraries repo and moved this here. I think I conflated the two repositories. Honestly to me examples means more free-form code where libraries is the more official, blessed bits.

Thank you for your feedback!

[taherv]

There is a lot more code here than what should be in a "demonstration for an API". Also, why is there even a platform aspect here ? (I mean UNIX vs Windows).
A bare bones demonstration, I imagine, should be just Go code that ideally doesn't even specify a toolchain to use.

[akutz]

Re: platform, to what are you referring? If you mean the Makefiles it's because I am determining whether the artifact being built has an extension of ".exe".

And Go plug-ins are only supported on Linux which is why I called that out.

Thank you for your feedback.

I'm concerned there is confusion because I tried to provide an overview of different concepts and ideas -- examples. Also, Go isn't the only language capable of gRPC, so I don't want to assume that Go is a default choice.

I'm going to close this PR and submit things in smaller chunks since my attempt at providing value as discreet elements in a single PR has seemingly confused people -- unable to see the trees because of the forest in the way.