Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for gRPC protocol #1623

Merged
merged 63 commits into from
Oct 21, 2020
Merged

Support for gRPC protocol #1623

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
d1a6073
simple starting point to have grpc (unary) requests
rogchap Sep 3, 2020
5b52b07
add gRPC specific metrics and refactor js API
rogchap Sep 4, 2020
2410ef5
add gRPC status codes and use state Dialer
rogchap Sep 7, 2020
bb05b01
add support for TLS
rogchap Sep 7, 2020
be8f784
manage some of the errors and add the message JSON to the response ob…
rogchap Sep 7, 2020
989f90e
add support for custom tags, headers and timeout
rogchap Sep 8, 2020
e08853c
place client in it's own file
rogchap Sep 8, 2020
3d25f99
fix TLS issues and add a timeout on the connection
rogchap Sep 8, 2020
28d5a86
some basic testing of the grpc client
rogchap Sep 9, 2020
5c15a25
add some standard tags
rogchap Sep 9, 2020
f431c07
Add Headers/Trailers and seperate the Error message in the response
rogchap Sep 9, 2020
623dbbb
tests for request, headers, reponse, errors, res headers and trailers
rogchap Sep 10, 2020
f08a5bf
test for metrics an fix golint messages
rogchap Sep 10, 2020
070b37a
update unix package
rogchap Sep 10, 2020
614be83
fix unix vendoring
rogchap Sep 10, 2020
657db28
fix up the system tags test and deal with some of the lint issues
rogchap Sep 10, 2020
4ce280e
more linting "fixes"
rogchap Sep 11, 2020
8250192
run enumer for new system tags
rogchap Sep 11, 2020
c46e726
more linting errors
rogchap Sep 11, 2020
f0e9c9c
linter really does not like me
rogchap Sep 11, 2020
5f91c4c
update package to match master
rogchap Sep 11, 2020
907207f
Merge branch 'master' into grpc
rogchap Sep 11, 2020
8ed72d4
vendor clean up
rogchap Sep 11, 2020
eaeef08
hack to fix windows file not found error
rogchap Sep 11, 2020
61bd22c
throw on connection errors
rogchap Sep 15, 2020
0dc562b
small API naming change to align more with the gRPC info that is retu…
rogchap Sep 17, 2020
f01b8bc
opps; fix test
rogchap Sep 17, 2020
251d259
add time to the metrics
rogchap Sep 18, 2020
b48f7b1
add debug information if --http-debug flag is enabled
rogchap Sep 24, 2020
fa71eae
fix linting issues
rogchap Sep 24, 2020
539c70a
fix up vendoring and depcricate warnings
rogchap Sep 27, 2020
8b4d384
protoreflect does not like the v2 interface
rogchap Sep 27, 2020
14d81d4
add RPS Limit to Invoke RPCs
rogchap Sep 30, 2020
53695e8
remove grpc_reqs metric as it's obsolte with the trend metrics count …
rogchap Oct 14, 2020
cf59c0e
fix for default values and simplify Response object to not use goja v…
rogchap Oct 14, 2020
7cd996a
extend method info and align with grpc lib to include leading slash f…
rogchap Oct 14, 2020
0eb10a4
return error on unknown parameters and better timeout parsing
rogchap Oct 15, 2020
59e98f8
remove string builder for better readability.
rogchap Oct 15, 2020
41ff188
make the channel buffered so that we don't have a goroutine leak
rogchap Oct 15, 2020
1edd999
scope request tags to the context rather than the client
rogchap Oct 15, 2020
aa934bc
return errors on param errors rather than continue
rogchap Oct 15, 2020
281aa4a
Merge branch 'master' into grpc
rogchap Oct 15, 2020
8d47d2f
go mod tidy
rogchap Oct 15, 2020
d7dc6c1
update vendor modules - I wish this was auto on mod tidy
rogchap Oct 15, 2020
33654b1
fix up some linting errors
rogchap Oct 15, 2020
3fc6969
static check on jsonpb package
rogchap Oct 15, 2020
197a518
refactor to offical protobuf APIv2 and rename API to just invoke
rogchap Oct 15, 2020
28fc5d8
update vendor directory
rogchap Oct 15, 2020
9334e76
ingnore unresolvable files and simplify parsing
rogchap Oct 15, 2020
49ae763
better handling of complex protobuf definitions
rogchap Oct 16, 2020
4f1ca02
resolve the full import graph for the proto files
rogchap Oct 16, 2020
1f62ddf
Merge branch 'master' into grpc
rogchap Oct 18, 2020
ab54b18
sync vendor folder
rogchap Oct 18, 2020
bb31412
remove rpc type system tag and rps limit
rogchap Oct 18, 2020
32d1812
deal with prealloc lint check
rogchap Oct 18, 2020
6c48328
handle serialisation errors
rogchap Oct 18, 2020
74df86d
scope error handling
rogchap Oct 18, 2020
83c390b
add tests for debug stats and support v1 proto messages
rogchap Oct 19, 2020
0f4baf0
need the APIv1 to support v1 proto messages
rogchap Oct 19, 2020
f7482b5
gofumpt-ed
rogchap Oct 19, 2020
4c83ec6
address feedback from team
rogchap Oct 21, 2020
512d34f
Merge branch 'master' into grpc
rogchap Oct 21, 2020
29a4b25
update import to use protocols dir
rogchap Oct 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
10 changes: 7 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,15 @@ require (
github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7 // indirect
github.com/gin-gonic/gin v1.1.5-0.20170702092826-d459835d2b07 // indirect
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
github.com/golang/protobuf v1.4.2
github.com/golang/snappy v0.0.0-20170215233205-553a64147049 // indirect
github.com/google/go-cmp v0.5.1 // indirect
github.com/gorilla/context v0.0.0-20160226214623-1ea25387ff6f // indirect
github.com/gorilla/mux v1.6.1 // indirect
github.com/gorilla/websocket v1.4.2
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/influxdata/influxdb1-client v0.0.0-20190402204710-8ff2fc3824fc
github.com/jhump/protoreflect v1.7.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a grpc and protobuf noob, but while reading up on them it seems like the newer google.golang.org/protobuf` does actually support what protoreflect is used for here? More specifically parsing protobuf definition during runtime and creating messages out of those definitions. Given this and the fact the one is the "standard" implementation, can we actually use that one?

Maybe there are some additional problems that I am not aware of?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true for the dynamic calls, but unfortunately it does not have the ability to parse the raw *.proto files. Google maintain that the C++ protoc tools is used for the parsing so there are few Go alternatives and protoreflect is the most complete choice.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm ... I guess nobody has done one that just parsers to the standard lib v2 interface types(if that is possible at all)?

I guess this is the smaller of the big new dependencies that will be included, but still, I would prefer if would've liked it to use less ... if possible

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a parser that does just that; but it's not complete (other priorities) https://github.com/rogchap/protoparser

I opted for a production ready version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mstoykov @na--
So; to make sure we stay "up-to-date", I have refactored a lot of the code to use the Go Protobuf APIv2 and only use this library for the parsing only. This was everything is more standard and if we need to switch out the parser, that can be easily done.
As a side effect I've renamed the API from Client.invokeRPC to just Client.invoke; it's all RPC (duh) and this lines up with the grpc library under the hood (at the least the Go version)

github.com/julienschmidt/httprouter v1.1.1-0.20180222160526-d18983907793
github.com/kardianos/osext v0.0.0-20170510131534-ae77be60afb1 // indirect
github.com/kelseyhightower/envconfig v1.4.0
Expand Down Expand Up @@ -67,10 +69,14 @@ require (
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasttemplate v0.0.0-20170224212429-dcecefd839c4 // indirect
github.com/zyedidia/highlight v0.0.0-20170330143449-201131ce5cf5
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7
golang.org/x/text v0.3.3
golang.org/x/time v0.0.0-20170927054726-6dc17368e09b
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/genproto v0.0.0-20200903010400-9bfcb5116336 // indirect
google.golang.org/grpc v1.31.1
google.golang.org/protobuf v1.24.0
gopkg.in/go-playground/assert.v1 v1.2.1 // indirect
gopkg.in/go-playground/validator.v8 v8.18.2 // indirect
gopkg.in/guregu/null.v2 v2.1.2 // indirect
Expand All @@ -80,12 +86,10 @@ require (

replace (
github.com/davecgh/go-spew => github.com/davecgh/go-spew v1.1.0
github.com/golang/protobuf => github.com/golang/protobuf v1.0.0
github.com/stretchr/testify => github.com/stretchr/testify v1.2.1
github.com/ugorji/go => github.com/ugorji/go v0.0.0-20180112141927-9831f2c3ac10
golang.org/x/crypto => golang.org/x/crypto v0.0.0-20180308185624-c7dcf104e3a7
golang.org/x/net => golang.org/x/net v0.0.0-20181201002055-351d144fa1fc
golang.org/x/sys => golang.org/x/sys v0.0.0-20180312081825-c28acc882ebc
golang.org/x/text => golang.org/x/text v0.3.0
gopkg.in/yaml.v2 => gopkg.in/yaml.v2 v2.1.1
)
111 changes: 104 additions & 7 deletions go.sum
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 10 additions & 8 deletions js/modules/index.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
"github.com/loadimpact/k6/js/modules/k6/crypto"
"github.com/loadimpact/k6/js/modules/k6/crypto/x509"
"github.com/loadimpact/k6/js/modules/k6/encoding"
"github.com/loadimpact/k6/js/modules/k6/grpc"
"github.com/loadimpact/k6/js/modules/k6/html"
"github.com/loadimpact/k6/js/modules/k6/http"
"github.com/loadimpact/k6/js/modules/k6/metrics"
Expand All @@ -33,12 +34,13 @@ import (

// Index of module implementations.
var Index = map[string]interface{}{
"k6": k6.New(),
"k6/crypto": crypto.New(),
"k6/crypto/x509": x509.New(),
"k6/encoding": encoding.New(),
"k6/http": http.New(),
"k6/metrics": metrics.New(),
"k6/html": html.New(),
"k6/ws": ws.New(),
"k6": k6.New(),
"k6/crypto": crypto.New(),
"k6/crypto/x509": x509.New(),
"k6/encoding": encoding.New(),
"k6/http": http.New(),
"k6/metrics": metrics.New(),
"k6/html": html.New(),
"k6/ws": ws.New(),
"k6/protocols/grpc": grpc.New(),
}
Loading