gRPC performing at nearly 50% of HTTP equivalent #1846
Comments
|
Hi @thecodejunkie, thanks for reporting this and for the comprehensive code to reproduce it. You're right, this is a surprising difference and we'd also expect better performance from the gRPC test, especially since you're reusing the TCP connection. I took a look at it and didn't find anything immediately obvious. Some notes:
So we would need more time to dig into this and determine the root cause. Please give us a few days to investigate and discuss this internally. |
|
Hi @imiric Thank you for a great reply!
Yep. Unfortunately, the TCP connection reuse is a (temporary) workaround to a limitation with macOS that causes it to throw
Yes, I tried to offset that slightly by introducing JSON serialization, of the same data structure, in the HTTP endpoint
Noted. Fiber was chosen because of the increased performance over
Much appreciated! Please let me know if I can provide additional help or information |
|
Hi again! I spent some time looking deeper into this, and while I can't say I managed to find the root cause, there are some improvements we could make to minimize this difference. More notes:
So in conclusion, I'm not sure what we can improve on the k6 side besides exposing those gRPC/HTTP2 options. It's difficult to compare both directly with synthetic tests as the underlying transport is so different. Hope this is helpful and if anyone else has more gRPC/HTTP2 experience, please chime in. :) |
|
@imiric Hi! It just dawned on me that I did read your reply, but I never actually answered you 😱 I wanted to stop by and thank you for your very thorough investigation of this issue! So THANK YOU! 😄 |
|
No worries @thecodejunkie, hope it was useful. Since we'd like to address this issue eventually, and to allow someone with more gRPC/HTTP2 experience to comment, let's leave this open for now. We'll consider exposing these and other options to the client, but that's likely going to be in a separate issue, and is currently not on the short-term roadmap. |
na--
added
evaluation needed
|
Hello, |
|
@LaserPhaser, do you experience the same problems? If so, can you please share some details and benchmarks? Skimming over this issue, it was mostly left open for further discussion, not necessarily with the expectation that anything needed to be fixed here? And the previous discussion and investigation was a long time ago, there have been substantial changes in k6, Go and the underlying gRPC and HTTP libraries, so some fresh data would definitely be required before we actually do anything. |
|
Basically I have the same problem. ENV: K6 master (b60fe88) Scenario: I've tried to profile k6: |
|
Basically I think it would be good enough to create xk6 extension Will try to implement it till the EOW |
|
Hi @LaserPhaser, |
|
@codebien Sure, thank you. |
|
Context: I am making a simple benchmarking between REST and gRPC. I am runnning the test on 20 vu's for 4 mins total with some timing conditions (attached the k6 script) Test scenarios: k6, throttled network upto 100KbPS (using wondershaper) , server and client are hosted in aws The results are very astonishing to me, The rest-api is performing as expected with iteration count of ~16/s and latency of ~1700s (p95). But, the grpc have iteration count of ~1.5/s whereas the latency is around ~60ms. I have tested the same in a normal environment ( with no network throttling) i am having the same issue with iteration count of rest 3-4 times more than that of grpc. also the latency(p95) in REST(~19ms) is slightly better then gRPC(~21ms) I am unable to figure out why am i getting these result, i-> rest is outperforming grpc is it a k6 limitation or there something wrong in my scripts, I have searched a lot in internet but hard luck, Can you please guide me here? |
|
Hey @kiwi-waquar, please post your question to the Community Forum, it is the place for support. Please, share your code using a repository, a gist, or directly in the comment using the collapsable feature. |
|
hey @codebien Thanks for replying, I will surely post my question there. gRPC server code
import com.google.protobuf.Timestamp;
import io.grpc.Grpc;
import io.grpc.InsecureServerCredentials;
import io.grpc.Server;
import io.grpc.stub.StreamObserver;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
@Component
@Slf4j
public class TestServer {
private Server server;
private void start() throws IOException {
int port = 7861;
server = Grpc.newServerBuilderForPort(port, InsecureServerCredentials.create())
.addService(new LargeObjectTextCallImpl())
.build()
.start();
log.info("TestServer started, listening on " + port);
Runtime.getRuntime().addShutdownHook( new Thread() {
@Override
public void run() {
System.err.println("*** shutting down gRPC server since JVM is shutting down");
try {
TestServer.this.stop();
} catch (InterruptedException e) {
e.printStackTrace(System.err);
}
System.err.println("*** server shut down");
}
});
}
private void stop() throws InterruptedException {
if (server != null) {
server.shutdown().awaitTermination(30, TimeUnit.SECONDS);
}
}
private void blockUntilShutdown() throws InterruptedException {
if (server != null) {
server.awaitTermination();
}
}
public void serverRunner() throws IOException, InterruptedException {
final TestServer server = new TestServer();
server.start();
server.blockUntilShutdown();
}
static class LargeObjectTextCallImpl extends MeteoriteLandingsServiceGrpc.MeteoriteLandingsServiceImplBase {
@Override
public void getLargePayloadAsList(EmptyRequest emptyRequest, StreamObserver<MeteoriteLanding> responseObserver){
MeteoriteLanding meteoriteLanding = MeteoriteLanding.newBuilder()
.setFall("fell")
.setName("Adhi Kot")
.setId(379)
.setNametype("Valid")
.setRecclass("EH4")
.setMass(4239)
.setYear(Timestamp.newBuilder().build())
.setReclat(32.100000)
.setReclong(71.800000)
.setGeolocation(generateGeoLocation())
.build();
responseObserver.onNext(meteoriteLanding);
responseObserver.onCompleted();
}
private GeoLocation generateGeoLocation(){
return GeoLocation.newBuilder().setType("Point").build();
}
}
}REST healthcheck code
script for REST
import http from 'k6/http';
import {check} from 'k6';
import {Rate, Trend} from 'k6/metrics';
let apiSuccessRate = new Rate('API Success Rate');
let apiLatency = new Trend('API Latency');
export let options = {
stages: [
{duration: '1m', target: 20}, // Ramp up to 20 virtual users over 1 minute
{duration: '2m', target: 20},
{duration: '1m', target: 10},
],
systemTags: ['status', 'method', 'url', 'name'],
};
export default function () {
let url = 'http://aws-env/abc/healthcheck/performance';
let res = http.get(url, {
headers: {
'Content-Type': 'application/json',
'accept': 'application/json',
},
name: "API - Rest performance",
});
if(res.status !== 200){
console.log(url)
console.log(res.body)
}
check(res, {
'is status 200 for API': (r) => r.status === 200
});
apiSuccessRate.add(res.status === 200);
apiLatency.add(res.timings.duration);
}script for gRPC
import grpc from 'k6/net/grpc';
import {check, sleep} from 'k6';
import {Rate, Trend}from 'k6/metrics';
let apiSuccessRate = new Rate('API Success Rate');
let apiLatency = new Trend('API Latency');
export let options = {
stages: [
{duration: '1m', target: 20}, // Ramp up to 20 virtual users over 1 minute
{duration: '2m', target: 100},
{duration: '2m', target: 100},
],
systemTags: ['status', 'method', 'url', 'name'],
};
const client = new grpc.Client();
client.load(['definitions'], 'test.proto');
export default () => {
client.connect('aws-env:443', {
});
let response = client.invoke('test.MeteoriteLandingsService/GetLargePayloadAsList', {});
check(response, {
'status is OK': (r) => r && r.status === grpc.StatusOK,
});
apiSuccessRate.add(response.status === grpc.StatusOK);
client.close();
};proto file
syntax = "proto3"; import "google/protobuf/timestamp.proto"; option java_multiple_files = true; package testLarge; service MeteoriteLandingsService { message EmptyRequest { message StatusResponse { message Version { message GeoLocation { message MeteoriteLanding { Benchmarking results
|
As stated by these two comments, I think most of the significant improvements possible here are around that, and potentially adding the equivalent version of If you, future reader, fall into this closed issue, and you think you're experiencing gRPC performance issues that are totally unrelated to what's discussed here, unexpected or where the potential improvement would come from a different part of the code, feel free to open a new issue with the specific details. Thanks! |
Hi,
I was chatting with @simskij, on the Gophers Slack about some performance observations I was having when using k6 to compare gRPC and HTTP endpoints (in the same service), and he suggested I post here as well to get some insights from the performance wizards :)
Both endpoints have a similar setup (for testing purpose) and for gRPC I get about 32k req/s and for HTTP I am seeing about 55k req/s. I was expecting gRPC to be a bit higher, but I might be wrong?
I run my service and k6 tests on the same machine, a 2.2Ghz 6 core i7 MBP with 32GB RAM. Below are the test results
Environment
Expected Behavior
I was expecting gRPC to be on par (or even faster) then my HTTP endpoint? @simskij was seeing similar results on his MBP, but wasn't 100% sure of the difference in results (i.e if it was to be expected and why, or if something other was at play)
Actual Behavior
gRPC performs at (almost) 50% of the HTTP endpoint
Steps to Reproduce the Problem
I've included all the code to reproduce this
main.go
chat.proto
chat.pb.go
grpc.js
http.js
The text was updated successfully, but these errors were encountered: